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interface  and  post  analysis  support  methods,  and  they  describe  the  basic  features  and 
distinctive  characteristics  of  each  product. 

The  capabilities  of  the  products  are  compared,  providing  the  reader  with  the 
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the  products  and  the  evaluation  criteria  provides  a  quick  overview  of  the  analysis. 
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EXECUTIVE  SUMMARY 

The  drawdown  and  budget  cutbacks  have  created  a  situation  where  the  military 
must  use  alternate  means  to  evaluate  existing  and  emerging  communications,  computer 
and  information  systems.  It  is  no  longer  feasible  to  build  computer,  communications  and 
information  systems  solely  for  testing  and  evaluation  purposes.  Money,  equipment, 
soldiers  and  time  are  not  available  to  disconnect  existing  information  systems  for  testing 
and  evaluation. 

It  is  standard  practice  to  utilize  software  simulation,  including  hardware  and  man- 
in-the-loop,  to  expedite  development,  testing,  and  evaluation  of  computer, 
communication,  and  information  systems.  This  thesis  examines  the  current  state-of-the- 
art  modulation  simulators,  so-called  because  signal  modulation  is  required  to 
communicate  information. 

Modulation  simulation  is  a  computerized  method  for  modeling  a  current  or  future 
communications,  computer,  or  command  and  control  system.  Modulation  simulators 
assist  the  user  in  designing,  evaluating,  testing,  and  reconfiguring  a  system.  It  is  much 
less  expensive  to  model  a  new  system  with  a  modulation  simulator  than  to  physically 
build  a  demonstration  example.  Modulation  simulators  allow  for  parameter  variation, 
protocol  variation,  load  testing,  and  reconfiguration  all  within  the  computer.  Modulation 
simulators  allow  users  to  examine  emerging  system's  interfaces  with  current  military, 
commercial  and  joint  networks,  and  they  provide  a  method  for  testing  futuristic 
technologies.  Additionally,  they  enable  the  user  to  evaluate  network  performance,  to  test 
the  effects  of  system  outages,  to  stress  the  system  under  peak  loading,  and  to  see  how  a 
network  would  react  as  equipment  is  moved,  simulating  the  flow  of  the  battle  or  OOTW 
(Operations  Other  Than  War). 

An  analysis  of  five  of  the  current  commercial  off  the  shelf  software  modulation 
simulator  products  provides  the  modeler  with  the  information  needed  to  make  an 
informed  product  selection.  The  five  products,  OPNET  Modeler  developed  by  MIL  3 
Inc.,  COMNET  HI  by  CACI  Products  Company,  Extend  created  by  Imagine  That  Inc., 
Workbench  by  Scientific  and  Engineering  Software  Inc.,  and  G2  from  the  Gensym 
Corporation,  were  evaluated  against  twenty-five  various  criteria.  The  following  table 
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provides  a  non-weighted  comparison/evaluation  of  the  products  and  some  general 
overview  information.  A  rating  of  1,  the  lowest,  to  5,  the  highest,  has  been  established 
for  each  evaluation  criteria. 


OPNET 
MODELER 

COMNET  HI 

EXTEND 

WORKBENCH 

G2 

General 
Usage 

Comm 

Networks, 

Computer 

Systems  and 

Applications, 

and  Distributed 

Systems 

Network 
Planning  Tool, 

LAN/WAN 

Simulation 
Tool 

System  Level 

Simulation 

Tool 

Intelligent  Real 
Time  Systems 

Vendor 

MIL  3  Inc. 

CACI  Products 
Company 

Imagine  That, 
Inc. 

SES 

Gensym  Corp. 

Platforms 

Sun  SPARC, 
DEC,  HP  UX, 
SGI,  Windows 

NT 

PCs  (Windows 
NT  and 
Windows  95), 
UNIX,  DEC, 
HP  UX,  SGI, 
ADC,  Solaris 

PCs  (Windows 
3.1,  NT  and 
Windows  95), 
Macintosh 

Sun  Micro- 
stations,  IBM, 
HP 

Digital,  IBM, 
HP,  Sun 
Micro-stations, 
PC's 

(Windows  NT 
and  Windows 
95) 

Price 

Commercial 
University 

$18,000.00 
$600.00 

$48,500.00 
$3000.00 

$695.00 
$350.00 

$23,500.00+ 
$500.00 

$40,800.00 
$500.00 

Accessibility 

3 

3 

3 

2 

3 

Completeness 

5 

3 

4 

4 

4 

Configuration 
Management 

5 

4 

4 

5 

5 

Cost 

4 

1 

5 

3 

2 

Data  Input 
Methods 

5 

5 

5 

5 

5 

Data 
Reduction 

5 

5 

5 

5 

3 

Documentation 

4 

3 

3 

4 

3 

Ease  Of  Use 

3 

4 

3 

3 

2 

Experimental 
Design 

4 

2 

5 

3 

3 

Extendibility 

4 

2 

4 

5 

5 

Extensibility 
Of  Results 

5 

2 

4 

4 

4 

Fidelity 

5 

4 

5 

5 

5 

Flexibility 

4 

4 

4 

4 

4 

Interoperability 

4 

2 

4 

4 

4 

Miscellaneous 

4 

2 

2 

2 

2 

Operational 
Environment 

5 

5 

5 

3 

5 

Post  Analysis 
Support 

5 

5 

3 

4 

5 

Reliability 

5 

5 

5 

5 

5 

Security 

3 

1 

1 

1 

3 

Software 

3 

3 

3 

4 

4 

Xll 


OPNET 
MODELER 

COMNET  m 

EXTEND 

WORKBENCH 

G2 

Training  / 
Tutorial 

4 

4 

4 

3 

2 

Troubleshooting 
Assistance 

3 

3 

4 

5 

3 

User  Interface 

4 

5 

5 

4 

4 

Validation 

5 

5 

5 

5 

5 

Verification 

5 

5 

5 

5 

5 

Non  Weighted 
Totals 

106 

87 

100 

97 

95 

Each  of  the  five  products  analyzed  has  special  characteristics  and  qualities.  The 
products'  functions  and  capabilities  vary,  making  each  product  perfect  for  a  different 
scenario. 

CACI's  COMNET  DI  is  the  best  choice  for  modeling  a  wide  area  network  or  local 
area  network.  COMNET  EH  provides  all  the  familiar  computer  network  icons  in  a  user 
friendly  environment  to  enable  the  user  to  quickly  model  and  analyze  a  computer 
network.  COMNET  HI,  however,  does  not  model  communication  or  information 
systems.  Nor  does  COMNET  IE  model  the  innards  of  a  computer. 

MEL  3  Inc.  developed  OPNET  Modeler,  which  can  model  computers, 
communication  and  information  systems  from  the  network  through  the  process  and  state 
levels.  Numerous  libraries  are  available  to  assist  the  modeler.  OPNET' s  primary 
disadvantage  is  its  fixed  run  speed.  This  can  cause  extensive  time  delays  during  a 
simulation  run  of  a  detailed  model. 

SES'  Workbench,  like  OPNET,  can  be  used  to  model  communication,  computer, 
and  information  systems.  In  addition,  Workbench  can  be  used  to  model  numerous  other 
types  of  systems,  to  include  queues,  electric  flow,  water  pollution,  etc.  The  ability  to 
model  more  than  just  communication,  computer  and  information  systems  provide  an 
added  flexibility  to  the  modeler. 

Extend,  developed  by  Imagine  That  Inc.,  is  similar  in  its  capabilities  to 
Workbench,  in  that  it  can  also  model  any  type  of  system.  Extend,  however,  provides  a 
more  user  friendly  environment  for  the  modeler.  Besides  the  documentation's  ease  of 
use,  the  menu  setup  has  a  more  logical  flow.  Workbench,  through  the  use  of  sub- 
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modules,  can  model  to  a  smaller  level  of  detail  than  Extend,  and  Workbench  can  handle 
larger  sized  models. 

G2,  also  similar  to  Workbench  in  its  system  modeling  capabilities,  is  in  a  class  by 
itself.  G2  utilizes  an  object  oriented  modeling  environment  with  a  structured  natural 
language.  This  product  is  a  great  selection  for  a  beginner  or  non-programmer.  Object 
oriented  modeling  enables  the  modeler  to  design  the  level  of  fidelity  and  complexity  to  be 
modeled.  G2  includes  a  wide  range  of  knowledge  bases  or  libraries,  though  the  labeling 
is  puzzling.  G2/object  oriented  modeling  is  not  an  easy  product/modeling  method  to 
learn. 

In  making  a  selection,  it,  also,  is  essential  to  look  at  the  capabilities  and 
characteristics  of  the  modeler.  Programming  skills  may  influence  the  product  selection. 
All  five  of  the  products  are  icon  and  menu  oriented.  COMNET  HI  provides  the  smallest 
programming  interface,  and  G2  has  an  easy  to  learn  structured  natural  language.  OPNET, 
Extend,  and  Workbench  each  use  a  modified  version  of  the  C  language.  Workbench 
presents  the  user  with  the  largest  interface  with  the  model's  code,  and  provides  the  user 
with  the  most  programming  opportunities. 

The  user  must  take  into  consideration  the  various  characteristics  of  the  system  to 
be  modeled  and  the  capabilities  of  the  modeler.  This  combination  must  be  compared  to 
the  analysis  of  the  products  to  correctly  select  the  proper  software  modulation  simulator 
for  the  mission. 


xiv 


I.  INTRODUCTION 

A.         MILITARY  AND  MODULATION  SIMULATORS 

It  is  standard  practice  to  utilize  software  simulation,  including  hardware  and  man- 
in-the-loop,  to  expedite  development,  testing,  and  evaluation  of  computer, 
communication,  and  information  systems.  This  thesis  examines  the  current  state-of-the- 
art  modulation  simulators,  so-called  because  signal  modulation  is  required  to 
communicate  information. 

Modulation  simulation  is  a  computerized  method  for  modeling  a  current  or  future 
communications,  computer,  or  command  and  control  system.  Modulation  simulators 
assist  the  user  in  designing,  evaluating,  testing,  and  reconfiguring  a  system.  It  is  much 
less  expensive  to  model  a  new  system  with  a  modulation  simulator  than  to  physically 
build  a  demonstration  example.  Modulation  simulators  allow  for  parameter  variation, 
protocol  variation,  load  testing,  and  reconfiguration  all  within  the  computer.  Modulation 
simulators  allow  users  to  examine  emerging  system's  interfaces  with  current  military, 
commercial  and  joint  networks,  and  they  provide  a  method  for  testing  futuristic 
technologies.  Additionally,  they  enable  the  user  to  evaluate  network  performance,  to  test 
the  effects  of  system  outages,  to  stress  the  system  under  peak  loading,  and  to  see  how  a 
network  would  react  as  equipment  is  moved,  simulating  the  flow  of  the  battle  or  OOTW 
(Operations  Other  Than  War). 

Most  organizations  currently  using  modulation  simulators  are  developmental, 
testing  and  evaluation,  or  acquisition  oriented.  Some  examples  include:  Directorate  of 
Combat  Development,  Fort  Gordon,  Georgia;  Joint  Interoperability  Testing  Command, 
Fort  Huachuca,  Arizona;  and  Defense  Information  Systems  Agency,  D8,  Arlington, 
Virginia. 

Modulation  simulators  could,  also,  be  utilized  by  tactical  operations  staffs. 
Networks  designed  for  tactical  exercises  could  be  modeled,  tested,  and  evaluated  for 
trouble  spots  prior  to  deployment  of  personnel  and  equipment.  Networks  could  be 
tweaked  for  optimal  reliability  prior  to  installation.  This  would  save  the  military  precious 
time  and  money. 


B.  SCOPE 

The  focus  of  this  thesis  is  on  commercial  modulation  simulators.  Budget  cutbacks 
have  reduced  military  software  development  resulting  in  most  units  turning  to 
Commercial  Off  The  Shelf  (COTS)  products. 

This  thesis  evaluates  five  of  the  current  commercial  modulation  simulators: 
OPNET  Modeler  3.0  by  MIL  3  Inc.;  COMNET  III  by  CACI  Products  Company;  Extend 
by  Imagine  That  Inc.;  Workbench  by  Scientific  and  Engineering  Software  Inc.;  and  G2  by 
Gensym  Corporation.  Each  product  is  evaluated  against  twenty-five  assessment  criteria. 
See  Appendix.  A  score  of  1  to  5  (5  is  the  best)  will  be  assessed  for  each  criteria.  The 
objective  is  to  provide  the  reader  with  the  information  required  to  intelligently  select  the 
most  useful  modulation  simulator  for  a  specific  mission. 

The  software  product  chapters,  Chapter  II  through  Chapter  VI,  include  an 
overview  of  each  product  and  a  detailed  evaluation  of  the  product  based  on  each  of  the 
criteria  and  sub-criteria.  Chapter  VII  is  the  analysis/comparison  chapter  which 
summarizes  the  results  and  provides  the  reader  with  a  basis  for  making  an  informed 
selection.  Examples  are  provided  in  Chapter  VH  to  assist  the  reader  with  how  to 
effectively  select  the  correct  software  product. 

Evaluations  are  based  on  a  hands-on  utilization  of  each  product,  tutorials,  world 
wide  web  and  user  demonstrations,  all  available  documentation,  and  interviews  with 
users,  sales  representatives,  and  technical  support  personnel. 

C.  PRODUCT  SELECTION 

The  software  products  selected  were  based  upon  availability  to  the  author,  current 
or  proposed  usage  in  the  military,  and  experienced  users'  recommendations.  Many  of  the 
products  can  be  used  to  model  a  wide  range  of  systems.  This  thesis  concentrates  on 
simulating  communication,  computer  and  information  networks.  This  includes  computer 
Local  Area  Networks  (LANs)  and  Wide  Area  Networks  (WANs),  current  military 
communication  systems  (Mobile  Subscriber  Equipment,  Digital  Group  Multiplexing 
Equipment,  SINCGARS,  etc.)  and  message  switching  networks.  Additionally,  the  thesis 


examines  the  software  product's  interface  capabilities  for  modeling  joint  and  emerging 
technologies. 


II.        OPNET  MODELER 

A.  INTRODUCTION 

OPNET  Modeler™  by  MIL  3  Inc.  is  a  computer  aided  engineering  system  for  the 
design,  simulation,  and  analysis  of  communication  networks,  computer  systems, 
applications,  and  distributed  systems.  OPNET  was  introduced  by  MIL3  Inc.  in  1987. 
Over  500  organizations,  both  civilian  and  Department  of  Defense,  are  currently  using 
OPNET  (MIL  3,  1987). 

OPNET  Modeler  has  a  detailed  object-oriented  environment.  It  utilizes  a 
windows  user  interface  with  icons,  drop  down  menus  and  text  editors  for  inputting  the 
network.  OPNET  provides  the  user  with  the  flexibility  to  model  top-down,  bottom-up  or 
middle-out. 

OPNET  provides  the  user  with  the  ability  to  model  from  the  Wide  Area  Network 
(WAN)  down  through  the  process  and  state  level.  The  process  and  state  level  is  used  to 
model  the  actions,  changes,  functions,  and  conditions  of  the  basic  components  of  a 
model.  Various  degrees  of  modeling  provides  the  user  with  the  added  flexibility  for 
modeling  current  or  designing  future  communication  and  computer  networks,  systems 
and  applications. 

The  following  information  is  gathered  from  the  UNIX  on-line  documentation,  the 
MIL  3  Inc.  world  wide  web  page,  the  MIL  3  Inc.  sales  pamphlets,  UNIX  on-line  tutorials, 
and  telephonic  interviews  with  sales  personnel. 

B.  ASSESSMENT  CRITERIA 

1.         Accessibility 

OPNET  Modeler  can  be  installed  on  these  platforms:  Sun  SPARC  Solaris  2X; 
Sun  SPARC  Sun  OS;  Digital  Equipment  Corporation  OSF/1;  Hewlett  Packard  UX; 
Silicon  Graphics  IRIX;  and  Intel  based  implementations  of  Windows  NT  3.5.1. 

OPNET  requires  75  megabytes  (MB)  of  disk  space  for  the  software  and  32  MB  of 
Remote  Access  Memory  (RAM).  MIL  3  Inc.  recommends  120  MB  of  disk  space  and  64 


MB  of  RAM.  OPNET  requires  access  to  an  ANSI  C  compiler,  which  is  included  with  the 
Solaris  and  Sun  packages. 

2.  Completeness 

OPNET  Modeler  is  a  comprehensive  product.  It  models  computers,  software, 
WANs,  Local  Area  Networks  (LANs),  data  channels,  or  any  other  systems,  applications 
or  processes  the  user  wishes  to  create.  It  includes  the  ability  to  model  current  and  future 
technologies. 

OPNET  includes  these  protocol  libraries:  Ethernet,  Token  Ring,  Fiber  Distributed 
Data  Interface  (FDDI),  Asynchronous  Transfer  Mode  (ATM),  Frame  Relay,  X.25,  Link 
Access  Procedure-Balanced  (LAPB),  Transmission  Control  Protocol  (TCP),  Internet 
Protocol  (IP),  User  Datagram  Protocol  (UDP),  Routing  Information  Protocol  (RIP), 
Distributed  Queue  Dual  Bus  (DQDB),  High  Speed  Serial  Bus  (HSSB),  In-vehicle 
Communication  Bus,  Advanced  Mobile  Phone  Service.  Connectionless  Network 
Protocol  (CLNP),  Transport  Protocol  4  (TP4),  and  Global  System  for  Mobile 
Communications  are  all  available  from  third  parties. 

OPNET  models  multiple  client/server  applications  (email,  database,  file  transfer, 
etc.),  peer-to-peer  protocol  delays,  server  backlogs,  and  transaction  response  time  and 
throughputs.  It,  also,  models  satellite  systems  and  mobile  communications  nodes. 

Some  military  system  libraries  are  available  for  this  product.  These  are  available 
through  various  Department  of  Defense  agencies. 

3.  Configuration  Management 

The  user  is  responsible  for  managing  input  and  output  files.  Separate  probes  will 
have  separate  output  files.  However,  a  rerun  of  a  model  will  overwrite  existing  output 
files. 

The  user  can  graphically  display  the  Network,  Node,  or  Process  levels  of  the 
model.  Each  layer  provides  the  user  with  a  different  degree  of  the  model's  detail. 

4.  Cost 

OPNET  Modeler  is  available  commercially  for  $18,000.00.  This  includes  all  of 
the  current  commercial  libraries  and  90  days  of  maintenance  and  technical  support. 


Additional  technical  and  upgrade  support  is  available  for  $3,000.00  per  year.  The 
university  edition  is  available  for  $600.00  per  year. 

MIL  3  Inc.  offers  a  free  one  day  training  seminar.  They,  also,  have  a  three  day 
OPNET  Modeler  course  available  in  Washington,  DC  for  $1,500.00.  Customized 
training  can  be  requested  at  the  user's  location.  The  pricing  of  this  option  is  dependent 
on  the  location,  facilities,  and  instruction  requested. 

5.  Data  Input  Methods 

OPNET,  in  addition  to  using  predefined  Generator  Modules  (Ideal  and  Clock), 
uses  stochastic  traffic  sources,  based  on  a  user  selected  probability  function,  for  creating 
message  inter-arrival  times,  and  uses  deterministic  sources  to  specify  exact  times  for 
message/packet  generation.  A  regenerative  source  can  be  used  to  create  "bursty" 
transmissions.  Tables  are  used  to  prompt  the  user  for  data  through  out  the  development  of 
the  model. 

External  Model  Access  includes  the  capability  to  extract  data  from  another  model. 
(See  Interoperability  for  additional  information) 

6.  Data  Reduction 

OPNET  Modeler  provides  immediate  accessibility  to  probe  (data  collection 
method)  results.  This  data  can  be  formulated  into  graphs  and  reports.  The  probes  are  user 
selected  and  include  node  statistics,  link  statistics,  global  statistics,  and  simulation 
attributes. 

The  following  probability  distributions  are  available  in  OPNET  Modeler: 

•  Bernoulli  •     Erlang  •     Poisson 

•  Chi-Squared  •     Exponential  •     Uniform 

•  Constant  •     Normal  •     Uniform-Integer 
These  can  each  be  utilized  via  menu  options. 

7.  Documentation 

On-line  documentation  is  included  with  the  UNIX  packages.  Hard  copy 
documentation  can  be  purchased  for  $500.00  a  copy. 


The  documentation  is  very  detailed  and  informative.  It  includes  a  comprehensive 
tutorial,  a  user's  manual,  two  volumes  on  modeling,  and  two  volumes  of  the  OPNET 
Modeler  simulation  kernels. 

No  design  specifications  were  available. 

8.  Ease  Of  Use 

This  product  is  fairly  complicated  to  the  new  user.  OPNET  Modeler  has  such 
detailed  capabilities  it  is  difficult  for  an  inexperienced  user  to  quickly  grasp  all  of  its 
qualities. 

OPNET  Modeler  enables  the  user  to  model  events  at  the  process  and  state  level. 
The  user  continues  with  the  nodal  model  and  then  integrates  it  all  together  in  a  network. 
This  provides  the  user  with  very  powerful  capabilities.  The  user  can  vary  the  degree  of 
depth  of  the  programming  parameters  and  events.  The  user,  also,  has  access  to  the 
entrance  and  exit  criteria  of  the  states.  The  process  and  state  level  is  the  most  basic 
breakdown  of  an  element  of  a  system.  This  enables  the  user  to  establish  the  specific 
conditions  required  to  enter  or  exit  a  state  of  a  system.  This  access  is  entirely  in  Proto- 
C™,  a  computer  language  very  similar  to  C. 

The  manuals  are  extremely  helpful  but  are  oriented  at  the  programmer  level. 
Many  modeling  concepts  are  just  assumed. 

Set  up  and  run  time  depend  on  the  model.  The  basic  models,  only  a  few  nodes, 
contained  in  the  tutorial  required  between  one  and  three  hours  of  setup.  Each  of  these 
only  took  a  few  minutes  to  run. 

It  is  difficult  to  adjust  the  speed  of  the  simulation  clock.  OPNET  is  designed  to 
run  as  fast  as  it  can.  The  total  execution  time  is  dependent  on  the  total  number  of  events, 
the  complexity  of  each  node  and  state,  and  the  type  of  platform  the  simulation  is  being 
executed  on.  The  user  can  speed  up  the  simulation  slightly  by  leaving  out  the  debugging 
option  during  the  execution.  Also,  the  simulation  can  be  forced  to  run  at  real  time  by 
establishing  a  comparison  of  the  game  clock  with  the  real  clock. 

9.  Experimental  Design 

This  product  allows  the  user  to  execute  iterative  runs.  Various  probes  can  be  set 
up  for  each  of  the  runs.  Separate  output  data  files  must  be  established  prior  to  the 


execution.  State  and  process  level  modeling  enables  the  user  to  develop  models  of 
emerging  technologies. 

10.  Extendibility 

OPNET  can  easily  be  modified.  The  user  must  first  access  the  desired  level  to  be 
altered,  then  right  click  on  the  item  to  be  altered.  This  provides  the  user  with  the 
parameter  menu.  Additionally,  OPNET  offers  the  option  of  an  attributes  menu.  This 
menu  will  allow  the  user  to  input  specific  numbers  immediately  prior  to  the  run's 
execution. 

OPNET  Model  uses  Proto-C™  coding.  Immediate  access  by  the  user  to  the  entire 
model's  code  is  not  available.  The  user  has  open  access  to  the  entrance  and  exit  coding 
for  the  state  diagrams.  Also,  if  an  error  occurs  during  compiling  or  execution,  a  text 
editor  with  the  subroutine  containing  the  error  will  be  accessible  to  the  user. 

1 1.  Extensibility  Of  Results 

OPNET  Model's  ability  to  model  at  the  state  level  enables  the  user  to  literally 
create  any  communication  or  computer  system..  Traffic  flow,  architectures,  software, 
protocols  and  data  channels  can  be  modeled.  This  program  is  limited  by  the  user's 
creativity,  modeling  and  programming  skills.  Communication  and  computer  icons  can  be 
modified  to  model  alternate  architectures. 

12.  Fidelity 

This  product  provides  results  consistent  with  the  input  parameters  and  models. 
Models  can  be  developed  with  various  degrees  of  detail. 

13.  Flexibility 

Nodes  can  simulate  movement  and  outages  during  run  execution.  Outages  can  be 
triggered  by  counters,  probability  distributions  or  the  game  clock.  Movements  can  be 
simulated  via  timed  outages. 

Once  the  network  is  modeled  it  takes  little  time  to  make  modifications. 

14.  Interoperability 

OPNET  has  an  External  Model  Access  (EMA).  EM  A  is  a  technique  of  accessing 
a  model  external  to  the  OPNET  program  (MIL  3,  1989-1996,  pp.  EMA-3).  This  includes 


creating  a  model,  modifying  a  model,  or  just  extracting  data  from  a  model.  This  is  all 
supported  via  C  programming  language  functions. 

OPNET,  also,  has  an  External  Compiler  Interface  (ECI).  ECI  permits  the 
interfacing  with  third  party  compilers  and  linker  programs. 

15.  Miscellaneous 

Map  overlays  are  available.  However,  they  only  provide  a  concept  for  the 
geographical  locations  of  WANs,  satellite  equipment  and  footprints,  and  mobile  cell 
layouts.  There  is  no  integration  of  equipment  with  terrain  density  or  elevation. 

Outages  can  be  implemented  via  triggers,  probability  distributions  or  timed 
events.  These  can  be  programmed  to  be  permanent  or  merely  timed  events. 

OPNET  can,  also,  model  atmospheric  propagation  among  satellite  links,  and 
airborne  communication  systems. 

16.  Operational  Environment 

The  portability  of  the  product  is  dependent  on  the  computer  platform.  All 
platforms  available  are  COTs. 

17.  Post  Analysis  Support 

OPNET  Modeler  provides  a  wide  variety  of  reports  and  graphs.  The  data 
collected  is  user  selected  via  probes.  Probes  are  established  to  collect  data  on  various 
points  in  the  network.  Multiple  probes  can  be  utilized  during  the  same  run.  Probe  types 
are:  node  statistics,  link  statistics,  global  statistics,  and  simulation  attributes. 

The  user  can  view  the  probe  results  with  multiple  or  single  output  vectors.  These 
vectors  are  selected  by  the  user  and  displayed  as  traces,  or  graphs.  OPNET  Modeler's 
analysis  tool  supports  a  variety  of  calculations:  histograms,  probability  distribution 
functions,  cumulative  distribution  functions,  and  confidence  intervals.  Additionally,  the 
user  can  create  mathematical  filters  to  help  evaluate  the  data. 

This  program  enables  the  user  to  export  the  results  with  encapsulated  Postscript 
files  or  TIFF  files  to  desktop  publishing  packages. 
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18.  Reliability 

OPNET  Modeler  is  a  Proto-C  based  product  with  icon  and  menu  user  interfaces. 
It  executes  in  its  own  run  time  environment.  Repeat  runs  produce  consistent  results. 

19.  Security 

OPNET  provides  a  restricted  access  security.  Three  types  of  users  can  be 
established.  A  basic  user  can  run  a  model,  but  can  not  view  the  model  in  OPNET  Editor. 
An  inspector  has  basic  user  permissions  plus  the  ability  to  "inspect"  the  model  with 
certain  interface  procedures.  A  full  user  has  complete  access  to  the  model  and  can  edit 
the  model  via  the  OPNET  Editor. 

20.  Software 

OPNET  Modeler  is  an  object-oriented  program  introduced  by  MIL  3  Inc.  in  1987. 
The  source  software  is  unavailable  to  the  user.  The  user  can  program  entrance  and  exit 
criteria  for  states  with  Proto-C.  Proto-C  is  very  similar  to  the  C  language.  The  user  is 
provided  access  to  some  programming  code  in  the  event  of  an  error.  This  is  only  to  assist 
the  user  in  troubleshooting  the  fault. 

21.  Training  /  Tutorial 

The  tutorial  is  comprehensive  and  informative.  It  takes  the  user  step  by  step 
through  the  menus  and  model  development  of  various  types  of  networks.  Some  of  the 
menus  were  incorrectly  labeled  in  the  tutorial,  but  the  correct  menu  was  pictured  and  the 
correct  steps  to  reach  the  menu  were  given.  The  tutorial  provides  the  user  with  a 
intermediate  knowledge  of  the  product.  Someone  with  little  or  no  modeling  background 
would  have  a  difficult  experience  using  the  product  after  the  tutorial.  It  is  written  for  a 
semi-experienced  modeler/programmer. 

22.  Troubleshooting  Assistance 

Troubleshooting  is  difficult.  All  troubleshooting  is  oriented  to  the  programmer. 
The  documentation  provides  a  basic  overview  and  helps  the  user  to  locate  the  cause  of  the 
fault.  Errors  are  listed  by  line  of  code,  which  is  immediately  displayed  for  the  user. 
Various  options  are  available  within  the  animation  probe  to  enhance  the  user's 
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troubleshooting  capabilities.  Assistance  is  available  by  phone,  email,  and  through  the 
world  wide  web. 

23.  User  Interface 

The  user  interface  is  a  mixture  of  windows,  menus,  icons,  and  text  editors.  A 
window  listing  the  current  functions  of  the  mouse  buttons  is  provided  at  the  bottom  of  the 
screen.  A  command  window  informs  the  user  of  the  current  status  of  the  screen. 
Multiple  levels  can  be  open  at  the  same  time,  and  a  recirculate  button  permits  the  user  to 
move  freely  from  level  to  level.  The  numerous  menus  can  be  confusing  to  novice  users. 

24.  Validation 

The  tool  is  capable  of  providing  a  very  detailed  solution.  The  simulation  of 
existing  systems,  where  measurement  data  was  available,  produced  results  consistent  with 
the  real  data. 

25.  Verification 

The  tool  seemed  to  function  as  described  in  the  documentation.  Instructions 
detailed  in  the  tutorial  produced  the  desired  results. 

C.        CONCLUSIONS 

OPNET  Modeler,  while  not  for  the  beginner,  is  a  comprehensive  tool.  It 
provides  the  user  with  the  ability  to  model  at  the  state  and  process  level.  This  provides  a 
greater  flexibility  in  the  modeling  of  new  technologies. 

OPNET  is  a  sophisticated  program  for  modeling  and  simulating 
communication  systems,  protocols,  and  networks.  OPNET  provides  a  friendly 
environment  for  building  and  analyzing  communication  and  computer  networks. 
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III.       COMNET  III 

A.  INTRODUCTION 

COMNET  m™  from  the  CACI  Products  Company  is  a  network  planning  tool 
designed  with  an  object-oriented  environment  to  model  Local  Area  Networks  (LANs), 
Wide  Area  Networks  (WANs)  and  Metropolitan  Area  Networks  (MANs).  The  network 
is  created  graphically  and  requires  no  actual  code  programming  by  the  user. 

Network  technologies  and  traffic  files  can  be  imported  from  various  systems. 
Animation  provides  the  user  with  the  ability  to  monitor  the  flow  of  the  network.  Building 
blocks,  objects  a  user  is  familiar  with,  which  include  computer  nodes,  routers  and  links, 
may  be  edited  to  define  specific  characteristics  of  a  network. 

The  basic  steps  to  build  a  model  using  COMNET  EI  are  define  the  topology,  and 
establish  traffic  and  computer  loads.  The  network  is  then  verified  for  correctness  and  run. 
The  game  clock  and  the  run  duration  can  easily  be  altered  by  the  user.  Reports,  selected 
by  the  user,  are  automatically  generated  during  the  run. 

COMNET  m  allows  the  user  to  (CACI,  1997): 

•  Predict  end-to-end  delays,  throughputs,  and  utilization  of  links,  buffers  and 
processors 

•  Reproduce  random  and  bursty  traffic  patterns 

•  See  peaks  and  valleys  of  traffic  not  just  snapshots  and  averages 

•  Pinpoint  sources  of  delays  and  bottlenecks. 

The  following  information  is  gathered  from  the  CACI  Products  Company's  world 
wide  web  page,  COMNET  HI  Tutorial  (Sullivan,  1996),  and  telephone  interviews  with 
the  sales  and  support  offices. 

B.  ASSESSMENT  CRITERIA 

1.         Accessibility 

COMNET  IE  software  currently  is  available  for  Windows  95,  Windows  NT,  all 
major  UNIX  computers,  Silicon  Graphics  Inc.,  AIX,  Solaris,  Digital  Equipment 
Corporation,  and  Hewlett  Packard  UX.  The  software  package  is  20  megabytes  (MB)  for 
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the  UNIX  or  personal  computer  (PC)  platforms.  It  also  requires  64  MB  of  remote  access 
memory  (RAM)  for  the  UNIX  and  32  MB  for  the  PC. 

2.  Completeness 

COMNET  HI  uses  nodes  and  links  as  the  primary  model  building  devices. 
Processor  nodes  model  communication  processing  devices  through  time  delays 
simulating  the  execution  of  software  applications  and  the  processing  of  packets. 
Processor  nodes  simulate  file  read  and  writes  and  application  runs  through  time  delays 
and  "Wait  For"  commands. 

Router  nodes  model  routers,  bridges,  switches,  hubs,  and  other  devices  that  have  a 
common  backplane  for  moving  packets  between  port  buffers. 

Switch  nodes  model  switches,  routers,  hubs,  and  other  communication  devices 
that  have  insignificant  packet  moving  delays. 

The  following  links  can  be  modeled  by  COMNET  m  (CACI,  1997,  pp.  "A  Quick 
Look  At  COMNET  m"): 

•  Aloha  •     CSMA  •     FDDI 

•  Priority  FDDI  •     Polling  •     DAMA 

•  Point-to-Point  •     Token  Ring  •     CSMA/CA 

•  Token  Passing  •     CSMA/CD 

The  Point-to-Point  links  include  ISDN  and  SONET  libraries. 

WANs  and  LANs  are  easily  modeled  through  use  of  subnets  and  WAN  cloud 
icons.  These  provide  the  user  with  various  views  of  the  network  hierarchy.  WAN  clouds 
can  be  used  to  model  frame  relay,  cell  relay  (ATM),  and  packet  switching  (X.25). 

Inter-machine  communications  can  be  modeled  if  the  user  knows  the  parameters 
of  the  interface.  COMNET  m  does  not  model  communication  system  networks. 

COMNET  HI  currently  does  not  model  any  military  specific  equipment. 

3.  Configuration  Management 

CACI  Products  Company  began  developing  simulations  over  35  years  ago. 
COMNET  HI  is  only  a  few  years  old  but  is  widely  used  throughout  the  civilian 
community.  COMNET  HI  is  currently  on  Version  1.3.  It  is  configured  as  a  proprietary 
execution  environment. 
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Users  are  responsible  for  managing  traffic  input  and  data  output.  Output  files  will 
be  overwritten  during  a  re-run  of  a  modified  model.  The  entire  network  is  displayed  for 
the  user.  Details  of  individual  nodes  can  be  accessed  and  parts  of  the  network  can  be 
"hidden"  via  COMNET  WAN  Clouds  and  Transit  Nets.  COMNET  WAN  Clouds  and 
Transit  Nets  are  icons  which  represent  portions  of  the  entire  network  to  reduce  the  clutter 
on  the  display. 

4.  Cost 

COMNET  m  is  $48,500.00  for  the  commercial  version,  $2,000.00  for  a  single 
user  university  edition,  and  $3,000.00  for  a  university  department  edition.  Training  costs, 
and  maintenance  and  upgrade  costs  for  the  first  year  are  included.  Afterwards,  the  yearly 
commercial  maintenance  and  upgrade  fee  is  $5,100.00  and  the  university  fee  is  $500.00. 

Additional  packages,  COMNET  Baseliner  and  the  COMNET  Advance  Features 
Package,  are  $9,000.00  each. 

5.  Data  Input  Methods 

Users  can  create  discrete  data  and  continuous  random  variable  data  through  tables. 
The  discrete  data  is  based  on  inputs  from  the  user  and  a  step  distribution.  The  continuous 
data  is  based  on  a  user  selected  probability  distribution  function.  COMNET  m,  also, 
features  source  objects:  Traffic,  Message,  Response,  and  Session.  Traffic  Source  is  used 
to  model  a  single  traffic  command.  Message  Source  works  with  a  transport  command 
and  models  protocol-control  messages.  Response  Sources  generate  replies  or 
acknowledgments;  and  Session  Sources  are  for  modeling  multiple  message,  bursts,  or 
virtual  circuit  routed  messages. 

Traffic  and  network  topologies  can  be  modeled  through  external  sources. 
TRAFLINK,  a  CACI  utility,  reads  external  traffic  files  and  formats  them  for  COMNET 
III  usage.  Network  topologies  can  be  extracted  from  Network  Management  Systems. 
Third  party  software,  Cabletron  Spectrum  topology  files,  HP  Open  View  topology  files, 
and  IBM  Net  View  6000  topology  files,  can  extract  an  existing  network's  topology  to  be 
imported  into  COMNET.  (CACI  Products  Company,  1995,  pp.  210) 


15 


6.  Data  Reduction 

COMNET  in  provides  immediate  graphs  and  reports.  A  Report  Menu  allows  the 
user  to  turn  on  and  off  various  reports  and  statistics  for  specific  items  (nodes,  links,  WAN 
clouds,  application  sources,  message  and  response  sources,  session  sources,  transport 
commands,  and  setup  commands).  Upon  completion  of  the  run  the  user  can  immediately 
"browse"  the  results. 

The  following  probability  distributions  are  available  in  COMNET  UL 

•  Beta  •     Geometric  •     Poisson 

•  Erlang  •     Hyper  exponential  •     Triangular 

•  Exponential  •     Log  normal  •     Uniform 

•  Gamma  •     Normal  •     Weibull 
Each  of  these  is  available  via  a  drop  down  menu  in  the  parameters  windows. 

7.  Documentation 

COMNET  IH's  documentation  consisted  of  one  book.  It  is  oriented  towards  the 
network  manager.  It  does  a  great  job  of  explaining  the  functions  and  capabilities  of 
COMNET  HI.  However,  it  does  not  describe  how  to  use  COMNET  BEL  It  is  left  up  to  the 
user  to  learn  how  to  use  the  product. 

A  tutorial  (Sullivan,  1996)  was  developed  by  a  third  party.  It  provides  step-by- 
step  instructions  on  how  to  use  the  major  features  of  COMNET  IE. 

The  lack  of  documentation  did  not  hamper  the  author's  ability  to  quickly  learn  and 
use  the  basic  features  of  the  product.  The  menus  and  icons  provide  the  user  with  an 
extremely  user  friendly  flow  for  developing  a  model. 

8.  Ease  Of  Use 

The  product  is  very  easy  to  use.  Network  topologies  are  quickly  developed  using 
drag  and  drop  icons  and  a  menu  driven  environment.  Parameters  are  altered  by  double 
clicking  on  the  various  network  objects.  Menu  options  enable  the  user  to  alter  the  game 
clock  speed,  animate  the  run,  trace  the  run  events,  and  run  iterative  trials.  Results  are  easy 
to  view  via  the  "Browse"  option  in  the  reports  menu.  The  network  set  up  time  is 
primarily  restricted  by  the  network  knowledge  of  the  user. 
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The  personnel  at  CACI  Products  Company  were  friendly  and  eager  to  assist.  The 
CACI  world  wide  web  page  (CACI,  1997,  pp.  "COMNET  Support")  provides  helpful 
email  addresses  and  phone  numbers.  COMNET  in  and  generic  modeling  questions  can 
be  answered  via  a  customer  hotline  provided  to  valid  users.  Additional  modeling 
assistance  is  available  at  a  cost  from  the  CACI  Modeling  Group. 

The  product  is  very  easy  to  learn.  The  thesis  tutorial  developed  by  Jeffrey 
Sullivan  (Sullivan,  1996)  provides  a  quick  overview  of  all  of  the  basic  COMNET  HI 
features. 

The  run  times  of  this  product  vary  with  the  detail  of  the  simulation.  The  simple 
networks  designs,  created  using  the  tutorial,  executed  in  only  a  few  minutes.  A  detailed 
10,000  node  network  with  multiple  events  and  detailed  parameters  would  take  much 
longer. 

9.  Experimental  Design 

The  modeling  tool  provides  the  user  with  the  ability  to  execute  trials  iteratively 
during  a  single  run.  Additionally,  the  automatic  parameter  feature  enables  the  user  to 
vary  the  parameters  during  the  iterative  executions.  This  provides  the  user  with  a  faster 
method  for  evaluating  proposed  network  changes.  The  user  can  modify  existing 
parameters  to  examine  alternate  computer  networks,  but  COMNET  can  not  model 
communication  systems. 

10.  Extendibility 

The  program  itself  cannot  be  modified  or  extended.  The  source  code  is  not 
provided  to  the  user.  COMNET  HI  does  provide  the  user  with  a  high  level,  simple 
program  interface.  The  programming  is  similar  to  Basic.  It  provides  greater  flexibility  to 
the  user.  Counters  and  triggers  can  be  established  to  initiate  follow  on  events.  An 
example  would  be  when  the  utilization  of  a  link  is  greater  than  60%  switch  to  an  alternate 
link. 

11.  Extensibility  Of  Results 

COMNET  HI  provides  the  user  with  two  primary  methods  for  evaluating  an 
emerging  architecture.  First,  with  the  basic,  high  level  programming  option  which  can  be 
used  to  set  counters  and  triggers  to  evaluate  the  results  of  an  event;  second  through  the 
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use  of  transit  nets.  Transit  nets  provide  a  more  detailed  option  for  modeling  transport 
protocols.  It  enables  the  user  to  create  a  protocol  like  frame  relay  via  an  ATM  link. 

COMNET  HI  does  not  specifically  model  data  channels  or  software  applications. 

12.  Fidelity 

The  tool  reliably  provides  results  consistent  with  the  input  parameters.  COMNET 
Baseliner,  an  add  on  feature,  enables  the  user  to  model  existing  networks  with  realistic 
external  data.  The  detail  of  the  model  can  only  be  slightly  varied. 

13.  Flexibility 

Nodes  can  be  set  up  to  start,  stop,  or  model  a  delay  during  the  execution. 
Parameters  can  be  easily  altered  by  double  clicking  on  the  items  icon. 

14.  Interoperability 

COMNET  Baseliner,  an  add  on  feature,  enables  the  user  to  model  existing 
topologies  and  use  realistic  traffic  files.  Actually  topology  data  can  be  taken  via  a 
Network  Management  System,  like  HP  Open  View,  Network  General,  Cabletron 
Spectrum,  IBM  Net  View  for  AK,  and  Castlerock  SNMP,  and  imported  in  COMNET. 
Network  traffic  data  can  be  gathered  from  Network  General  Distributed  Sniffer  System, 
Frontier  Software  NETscout,  HP  NetMetrix,  CompuWare  EcoNET  and  3COM 
LANsentry.  This  data  can  then  be  imported  into  COMNET  HI  via  COMNET  Baseliner. 
This  importing  capability  provides  the  user  with  the  ability  to  model  a  current  network 
and  real  data  to  run  on  it.  This  model  can  then  be  tested  to  find  any  network  problems, 
evaluated  for  interfaces  with  new  equipment,  tweaked  to  find  the  peak  performance 
design,  etc. 

A  small  amount  of  traffic  gathered  can  be  emulated  to  create  a  large  database  of 
"realistic"  data.  This  prevents  the  user  from  having  to  gather  hours  of  real  data. 

15.  Miscellaneous 

Outages  can  be  caused  or  enabled  through  parameter  settings  and/or  by 
establishing  triggers  and  counters.  Outages  can  be  for  a  predefined  time,  simulating  a 
nodes  movement;  for  a  random  time,  simulating  a  maintenance  outage;  or  permanently,  to 
simulate  equipment  destruction. 
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Frequency  and  terrain  options  are  not  available  with  COMNET  EDL 

16.  Operational  Environment 

This  product  is  available  for  Windows  95,  Windows  NT,  UNIX  stations,  SGI, 
ADC,  Solaris,  Dec,  and  HP  UX  systems.  All  these  systems  are  COTs  and  easily 
obtainable  to  users.  Portability  depends  on  the  platform  the  software  is  installed  on. 

17.  Post  Analysis  Support 

COMNET  m  provides  a  wide  variety  of  reports.  Each  of  these  is  available  for 
viewing  immediately  after  the  run  execution.  Reports  must  be  turned  on  by  the  user  for 
the  data  to  be  collected.  The  following  reports  are  available  within  COMNET  DI  (CACI, 
1997,  "COMNET  m  Features  List"): 

•  Node  Reports:  Processor  and  disk  utilization;  Received  message  counts; 
Session  level;  Call  counts;  Call  level;  Buffers  by  node;  Buffers  by  port 

•  Link  Reports:  Channel  utilization;  Utilization  by  application;  Utilization  by 
protocol;  Frame  Size;  Collision  Statistics;  Token  ring  statistics;  Session  level; 
Call  counts;  Call  level 

•  WAN  Cloud  Reports:  Message  and  Response  Source  Reports;  Message  delay; 
Message  delivered  delay;  Packet  delay 

•  Cell  Source  Reports:  Blocked  call  counts;  Disconnected  call  counts; 
Preempted  call  counts 

•  Transport  Command  Reports:  Message  delay;  Packet  delay 

•  Session  Source  Reports:  Message  delay;  Message  delivered  delay;  Packet 
delay;  Setup  delay;  Session  Length;  Setup  counts;  Transport  Layer 

•  Snapshot  Reports  Node  reports;  Link  reports;  Cloud  virtual  circuits 

•  Statistics  file  for  export  to  other  analysis:  Real-time  plots;  Post-processed 
plots;  Percentiles;  Raw  data  plot  file 

18.  Reliability 

The  product  is  coded  primarily  in  a  high  level,  icon  drag  and  drop  language.  It 
executes  in  its  own  run  time  environment.  Modeling  errors  may  cause  the  experiment  to 
suddenly  terminate.  Diagnostic  information  is  available. 
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19.  Security 

No  security  options  are  available. 

20.  Software 

CACI  Products  Company  has  been  a  simulation  leader  for  35  years.  COMNET  EI 
Version  1.3  was  fielded  in  December  1996.  The  source  software  code  is  unavailable  to 
the  user.  The  product  is  a  drag  and  drop  icon  and  menu  based  environment.  It  has  a 
simple  flow  for  network  development.  CACI  distributes  minor  releases  throughout  the 
year  and  major  releases  periodically.  Over  2000  various  universities  and  companies  are 
currently  using  COMNET  m  (CACI,  1997,  pp.  "A  Quick  Look  at  COMNET  HI").. 
COMNET  IQ  is  an  object-oriented  software  application. 

21.  Training  /  Tutorial 

CACI  Products  Company  provides  a  free  three  day  overview/orientation  training 
to  new  users.  Additionally,  the  tutorial  (Sullivan,  1996)  is  a  great  beginners  manual. 
Training  is  primarily  at  the  beginner  level.  COMNET  IQ  is  not  a  difficult  program  to 
learn. 

22.  Troubleshooting  Assistance 

Animation  and  event  tracing  are  available  to  assist  the  user  in  network  flow 
monitoring  and  troubleshooting.  All  valid  users,  those  having  paid  the  yearly  upgrade 
fee,  are  provided  with  a  Customer  Support  Hotline.  CACI,  for  a  cost,  has  a  Modeling 
Group  to  assist  companies  in  developing  a  model  for  a  network.  Assistance  is  available 
via  email  and  the  world  wide  web. 

23.  User  Interface 

The  product  is  very  user  friendly.  The  network  topology  can  be  created  via  drag 
and  drop  icons  or  with  menu  options.  Once  the  topology  is  established,  parameters  may 
be  altered  by  double  clicking  on  the  items.  A  parameter  window  will  appear  with  drop 
down  menus  to  assist  the  user.  The  verify  option  provides  a  detailed  list  of  all  errors 
present  in  the  network  topology  or  basic  flow. 
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The  user  may  alter  the  run  time  parameters  (clock  speed,  run  iterations,  animation, 
event  trace)  through  a  simple  menu.  Post  analysis  support  allows  the  user  to  easily 
"browse"  the  reports  and  statistics. 

24.  Validation 

The  simulating  of  existing  networks  where  measurement  data  is  available 
produced  results  consistent  with  the  real  data. 

COMNET  HI  provides  the  user  with  a  basic  to  intermediate  analysis  of  the 
network  and  its  interfaces.  This  is  dependent  on  the  parameters  and  events  established  by 
the  user. 

25.  Verification 

Upon  completion  of  the  model,  the  product  does  a  quick  verification  of  the 
network  connections. 

The  tool  seemed  to  function  as  described  in  the  documentation.  Instructions 
detailed  in  the  tutorial  produced  the  desired  results. 

C.         CONCLUSIONS 

COMNET  III  is  a  simple,  straightforward,  comprehensive  network  planning  tool. 
User  training  is  easy,  and  icon-based  network  designing  allows  for  quick  construction  and 
reconfiguration.  COMNET  HI  provides  a  method  for  analyzing  and  optimizing 
computer-based  networks. 
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IV.       EXTEND 

A.  INTRODUCTION 

Extend,  developed  by  Imagine  That  Inc.,  is  an  advanced  simulation  tool  designed 
to  develop  dynamic  models  of  real-life  processes.  It  enables  the  user  to:  predict  the 
course  and  results  of  certain  actions;  understand  why  observed  events  occur;  identify 
problem  areas  before  implementation;  explore  the  effects  of  modifications;  confirm  that 
all  variables  are  known;  evaluate  ideas  and  identify  ineffectiveness;  gain  insight  and 
stimulate  creative  thinking;  and  communicate  the  integrity  and  feasibility  of  plans 
(Imagine  That  Inc.,  1995,  pp.  4). 

B.  ASSESSMENT  CRITERIA 

1.  Accessibility 

Extend  is  available  for  Windows  3.1  or  later,  Windows  95,  Windows  NT, 
Macintosh  6.7  or  later,  and  Power  Macintosh  System  7.  10  MB  of  hard  disk  space  are 
required  for  the  Windows  version  and  8  MB  of  hard  disk  space  are  required  for  the 
Macintosh  version.  Both  platforms  require  4  MB  of  RAM  but  8  MB  are  recommended. 

Models  built  on  either  platform  can  be  run  on  the  other  platform.  For  example,  if 
you  build  a  model  or  library  on  a  Macintosh  computer,  it  can  easily  be  read  on  a 
Windows  computer. 

2.  Completeness 

Extend  uses  icons  or  building  blocks  to  build  models.  15  libraries  of  various 
building  blocks  are  included  with  Extend.  The  building  blocks  represent  the  smallest 
level  or  function  of  a  model.  The  user  must  build  computers,  communications 
equipment,  systems,  etc.  from  the  basic  functions  and  flows.  This  requires  the  user  to  be 
proficient  in  modeling. 

Users  can  create  new  icons  which  can  be  saved  in  the  libraries.  They  can  also 
save  models  as  templates  to  be  used  in  other  models.  This  provides  the  user  with  the 
power  to  create  the  computers  and  equipment  to  be  reused  in  later  models.  Additionally, 
a  list  of  libraries  available  through  third  parties  is  available  through  Imagine  That,  Inc. 


23 


3.  Configuration  Management 

Imagine  That  Inc.  was  founded  in  1987  to  market  Extend  software.  Extend  is 
current  on  version  3.2. 

The  user  is  responsible  for  managing  input  and  output.  Extend  permits 
hierarchical  modeling.  The  user  via  levels  (subroutines)  can  make  a  complex  system  easy 
to  build  and  easy  to  understand. 

4.  Cost 

Extend  is  available  for  $695.00.  The  university  price  is  $350.00.  Technical 
support  is  free  provided  the  user  has  purchased  the  most  recent  upgrade  version.  Today's 
current  upgrade  from  version  3.1  to  version  3.2  is  $99.00. 

Imagine  That  Inc.  does  not  offer  any  in  house  training  programs. 

5.  Data  Input  Methods 

Extend  includes  an  input  random  number  menu,  and  an  input  data  menu.  The  first 
provides  the  user  with  a  distribution  selection,  and  the  latter  permits  the  user  to  enter 
specific  data  values. 

Extend  uses  text  files  for  importing  and  exporting  data.  Text  files  can  be  created 
in  another  application,  like  a  spreadsheet,  word  processor  or  database,  then  read  into 
Extend,  and  vice  versa.  This  is  easier  for  inputting  lists  of  files  and  for  manipulating 
simulation  results. 

6.  Data  Reduction 

Plotters  are  built  into  the  model  to  gather  statistics.  Results  of  the  plotters  can  be 
immediately  displayed  on  the  screen. 

The  following  are  available  to  gather  statistical  information  from  the  results:  D- 
Binomial,  D-Exponential,  D-Gamma,  D-Log  Normal,  D-Pascal,  D-Poisson,  Gaussian, 
Mean,  Random,  Random  Get  Seed,  Random  Real,  Random  Set  Seed,  Standard  Deviation 
Population,  Standard  Deviation  Sample,  Statistical  Value.  (Imagine  That,  Inc.,  1995,  pp. 
287) 
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7.  Documentation 

Extend  has  one  book  for  its  documentation  (Imagine  That,  Inc.,  1995).  It  is  very 
comprehensive.  Chapters  1  through  3  consist  of  a  tutorial  of  Extend' s  basic  features. 
Chapter  4  is  an  overview  of  continuous  and  discrete  modeling.  Chapter  5  discusses  all  of 
Extend' s  libraries.  Chapter  6  describes  the  methods  to  get  data  into  and  out  of  Extend. 
The  rest  of  the  book  provides  a  more  in  depth  look  at  Extend,  modeling,  how  to  build 
icons,  and  a  list  of  common  errors  and  their  causes. 

8.  Ease  Of  Use 

Extend  is  fairly  easy  to  use.  Models  are  constructed  with  building  blocks  via 
windows  and  menus.  Double  clicking  on  a  building  block  provides  access  to  the  items 
attributes. 

Extend' s  biggest  challenge  is  the  user  must  know  the  intricate  details  of  the 
system  or  equipment  to  correctly  model  the  functions  and  flow. 

Set  up  time  is  dependent  on  the  depth  of  the  model.  The  basic  examples  provided 
in  the  tutorial  took  about  15  minutes  to  create  and  only  a  few  minutes  to  run. 

Extend  allows  the  user  to  vary  the  run  time,  and  the  step  speed  or  clock  speed. 
The  user  can,  also,  vary  the  model  while  it  is  running  online. 

9.  Experimental  Design 

Extend  supports  Monte  Carlo  and  batch  mode  simulations.  Iterative  runs  can  be 
conducted.  The  plotters  will  only  maintain  the  most  recent  four  plots.  Extend' s  basic 
features  enable  the  user  to  model  emerging  systems. 

10.  Extendibility 

Extend  primarily  uses  icons  or  building  blocks  to  construct  models.  Extend' s 
source  code  is  the  ModL  language.  It  is  based  on  the  language  C. 

Extend  permits  the  user  to  create  their  own  building  blocks  or  modify  existing 
ones.  The  user  can  use  "scripting"  to  design  code  for  the  model. 
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11.  Extensibility  Of  Results 

Extend  allows  the  user  to  create  building  blocks.  These  can  be  used  to  create 
virtually  any  communication  or  computer  system..  Extend' s  extensibility  is  limited  to  the 
modeling  and  programming  capabilities  of  the  user. 

12.  Fidelity 

The  tool  reliably  provides  results  consistent  with  the  input  parameters  and  the 
model.  The  user's  ability  to  create  building  blocks  could  effect  the  behavior  of  the 
model.  Models  can  be  developed  with  various  degrees  of  detail. 

13.  Flexibility 

Extend  provides  the  ability  to  alter  the  model  both  on  and  off  line.  The  user  can 
simulate  the  time  lost  for  the  movement  of  a  node  or  for  maintenance  down  time  through 
the  use  of  time  delays,  blocks,  and  triggers. 

14.  Interoperability 

Extend  has  cross-platform  capabilities  between  Windows  and  Macintosh.  Extend 
allows  users  to  copy  data  from  text  entry  fields  and  data  tables  into  other  applications. 

Extend  permits  the  user  to  create  building  blocks  to  communicate  with  scientific 
equipment  and  other  hardware  devices.  Extend,  also,  provides  two  methods  for  accessing 
code  written  in  another  language:  XCMD  (for  Macintosh)  and  DLL  (for  Windows). 

15.  Miscellaneous 

Outages  can  be  caused  or  enabled  through  delays,  triggers  and  blocks.  Outages 
can  be  for  a  predefined  time,  simulating  a  nodes  movement;  for  a  random  time, 
simulating  a  maintenance  outage;  or  permanently,  to  simulate  equipment  destruction. 

Frequency  and  terrain  options  are  not  available  with  Extend. 

16.  Operational  Environment 

This  product  is  available  for  Windows  and  Macintosh  systems.  All  of  these 
systems  are  COTs  and  easily  obtainable  to  users.  Desktop  and  notebook  computers  are 
both  very  portable. 
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17.  Post  Analysis  Support 

Extend  has  a  wide  variety  of  plotters  available  to  display  output:  Discrete  Event 
(DE)  Error  Bars;  DE  Multi-Simulation;  DE;  Error  Bars;  Fast  Fourier  Transform; 
Input/Output;  Multi-Simulation;  Histogram;  Scatter;  Scatter  (4);  Strip;  and  the  Worm. 

The  user  can,  also,  select  blocks  to  record  data  from  for  a  report.  These  reports 
only  display  final  simulation  values. 

18.  Reliability 

Repeated  runs  of  models  based  on  constant  inputs  produce  the  same  results. 
Extend  executes  in  its  own  run  time  environment. 

19.  Security 

No  security  features  are  available. 

20.  Software 

Imagine  That  Inc.  developed  Extend  in  1987.  Extend  is  written  in  ModL,  a  C 
based  language.  The  user  creates  models  by  connecting  building  blocks  to  simulate  the 
basic  functions  and  flows  of  the  system. 

21.  Training  /  Tutorial 

Imagine  That  Inc.  does  not  offer  any  in  house  training  courses.  There  are  some 
third  party  training  courses  available  for  a  cost. 

The  first  three  chapters  of  the  Extend  users  manual  are  a  comprehensive  tutorial. 
They  provide  the  user  with  a  complete  overview  of  Extend' s  major  features. 

22.  Troubleshooting  Assistance 

Animation  and  event  tracing  are  available  to  assist  the  user  in  network  flow 
monitoring  and  troubleshooting.  Common  error  messages  and  their  causes  are  listed  in 
an  appendix. 

Extend  provides  phone,  email,  fax,  and  world  wide  web  technical  assistance.  Free 
technical  assistance  is  available  to  all  users  modeling  with  the  latest  Extend  upgrade. 
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23.  User  Interface 

The  product  is  very  user  friendly.  Extend' s  user  interface  is  through  building 
blocks  and  menus.  The  user  can  quickly  and  easily  build  a  model  by  selecting  building 
blocks  from  the  menu  of  libraries,  and  then  connecting  the  blocks  together. 

24.  Validation 

Extend' s  degree  of  accuracy  is  based  on  the  users  design.  A  properly  designed 
model  produces  realistic  results. 

The  user  creates  the  model  to  the  depth  of  answer  required. 

25.  Verification 

Extend  will  verify  the  basics  of  the  code  during  compilation. 
The  tool  seemed  to  function  as  described  in  the  documentation.  Instructions 
detailed  in  the  tutorial  produced  the  desired  results. 

C.        CONCLUSIONS 

The  program  Extend  is  not  difficult  to  learn.  The  user  must  be  able  to  model  the 
basic  functions  and  flows  of  each  element  and  combine  them  to  build  the  model. 

Due  to  Extend' s  purpose  of  modeling  real  life  processing  in  a  wide  variety  of 
areas,  many  of  its  features  are  generic  and  not  communication  and  computer  specific. 
This  does  not  restrict  the  program's  ability  to  model  communication  and  computer 
networks  but  instead  expands  the  user's  modeling  options. 


28 


V.        WORKBENCH 

A.  INTRODUCTION 

Workbench  developed  by  Scientific  and  Engineering  Software,  Inc.  (SES)  is  a 
general  purpose  modulation  and  simulation  tool.  Workbench  is  a  collection  of  software 
tools  used  to  design  and  evaluate  sophisticated  systems  of  various  types.  It's  major 
components  are:  SES/design  ™,  a  graphical  editor;  SES/sim,  a  translation  and  simulation 
module;  and  SES/ scope,  an  animated  simulator. 

Workbench  is  an  object-oriented  product.  Models  are  built  with  a  top  down 
construction,  beginning  with  the  catalog  level,  then  the  module  and  sub-module  levels. 

Workbench  is  designed  to  model  the  essential  characteristics  of  a  system,  while 
ignoring  the  unimportant  details.  It  focuses  on  the  high-level  system  architectural  issues. 

B.  ASSESSMENT  CRITERIA 

1.  Accessibility 

Workbench  is  available  for  the  Sun,  IBM,  and  Hewlett  Packard  UNIX  platforms. 

2.  Completeness 

Workbench  consists  of  a  variety  of  module  and  sub-module  building  blocks/icons. 
These  building  blocks  can  be  used  to  create  models  of  current  and  future  computers, 
software,  and  communications  equipment.  The  models  are  constructed  by  connecting  the 
build  blocks  or  nodes  with  arcs.  Workbench  permits  the  user  to  create  icons.  These  icons 
must  be  based  on  a  basic  building  block  but  can  be  stored  in  a  library  for  future  usage. 

The  basic  building  blocks  or  the  icon  palettes  for  Workbench  are:  Resource 
Management  Nodes:  service,  delay,  resource,  allocate,  release,  create,  destroy,  block  and 
set;  Transaction  Flow  Nodes:  source,  sink,  fork,  join,  split,  loop,  interrupt,  resume,  and 
branch;  Sub-Module  Management  Nodes:  enter,  call,  sub-model,  and  return;  and 
Miscellaneous  Nodes:  user,  declaration,  and  super.  The  catalog  level  provides  the  user 
with  these  building  blocks:  main  module,  generic  module,  remote  module,  interface 
module,  object  file,  cosim  interface,  archive  file,  user  make  rules,  remote  catalog  and  a 
palette  icon. 
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Once  a  module  is  bui    it  can  be  archived  for  use  in  future  models.  This  enables 
the  user  to  create  a  library  of  frequently  used  items. 

Workbench  includes  a  variety  of  queuing  rules.  These  include  priority  rules 
which  allow  no  priority,  preemptive,  and  non-preemptive  priorities.  The  time  rules 
include:  first  come,  first  served;  last  come,  last  served  preemptive  resume;  processor 
sharing,  and  round  robin. 

3.  Configuration  Management 

SES,  founded  in  1971,  has  evolved  from  a  consulting  and  research  organization  to 
a  global  software  and  consulting  firm.  For  more  than  25  years  SES  has  been  providing 
simulation  technology  to  its  customers. 

The  user  is  responsible  for  managing  input  and  output  files.  User's  can  specify 
which  nodes  to  gather  statistics.  A  rerun  will  overwrite  previous  output  files. 

A  user  can  view  the  modules  and  sub-modules  developed  during  the  building  of 
the  model.  The  main  module  is  typically  designed  to  be  a  complete  diagram  of  the 
network  or  system  being  modeled. 

4.  Cost 

A  SES  sales  representative  will  help  design  a  software  package  of  Workbench  and 
its  associated  files  to  meet  the  users  needs.  Workbench's  starting  commercial  price  is 
$23,500.00.  The  university  price  is  $500.00.  The  government  price  is  20%  of  the  cost  of 
the  civilian  priced  package.  Technical  support  and  upgrade  costs  are  15%  of  the  original 
package  price. 

Training  is  available  at  SES  Regional  Sites  for  $8,000.00  per  person.  This  is  for 
the  three  day  introductory  course. 

The  Network  Framework  Library,  consisting  of:  X.25,  TCP,  ATM,  FDDI,  SNA, 
Client/Server  and  Ethernet,  can  be  purchased  for  $5,000.00  (civilian  price).  SES,  also, 
has  a  huge  list  of  partially  developed  models.  These  models  are  free  and  can  be  requested 
as  needed. 
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5.  Data  Input  Methods 

Users  can  input  data  internally  via  tables.  External  interfaces  include:  C  and  C++ 
programs  and  Objectbench  files.  The  users  must  establish  the  link  or  interface  between 
Workbench  and  the  external  functions. 

6.  Data  Reduction 

Workbench  provides  immediate  access  to  statistics  gathered  during  a  simulation 
run.  The  user  can  select  global  or  specific  statistics  to  be  gathered.  The  standard  statistics 
that  Workbench  gathers  are:  inter-arrival,  lifetime,  population,  q-population,  q-response, 
quantity,  response,  and  utilization.  Users  can,  also,  create  user-defined  "collect" 
statements  in  C  language  to  accumulate  additional  continuous  or  discrete  statistics  not 
ordinarily  collected  by  Workbench. 

Workbench  includes  these  probability  distributions:  empirical,  erlang, 
exponential,  gamma,  hyper-exponential,  normal,  triangular,  uniform,  binomial, 
geometric,  I-empirial,  I-uniform,  and  poisson. 

7.  Documentation 

Workbench's  documentation  is  quite  extensive.  The  Quick  Start  Manual  provides 
a  basic  overview  of  the  most  commonly  used  features  and  includes  an  example  or  mini 
tutorial  on  creating,  compiling  and  animating  a  simple  model.  Creating  Models, 
Simulating  Models,  SES/sim  Language  Reference,  Technical  Reference,  Query 
Interpreter  and  Language  Reference,  Cosim  Reference,  and  an  Installation  Manual  are, 
also,  included  in  the  Workbench  documentation. 

No  design  specifications  were  available. 

8.  Ease  Of  Use 

The  basic  features  of  Workbench  are  fairly  easy  to  learn  and  use.  Menus  and 
icons  create  a  user  friendly  environment  that  provides  the  user  with  a  graphical  interface 
to  modeling.  Workbench,  however,  is  not  designed  to  model  any  specific  type  of 
network.  It  is  a  general  modeling  tool.  What  this  means  is  the  libraries  and  icons  have 
basic  low  level  functions.  The  user  must  create  anything  specific  needed  for  the  model. 
The  user  needs  to  be  a  proficient  modeler  to  get  the  maximum  usage  of  this  software. 
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Workbench  uses  a  top  down  modeling  approach.  The  user  establishes  a  catalog 
with  a  main  module  and  possibly  some  other  generic  modules.  Each  module  is  then 
developed  further  as  a  sub-module.  The  user  can  "travel"  back  and  forth  between  the 
levels  of  the  modules  and  sub-modules.  This  level  approach  allows  the  user  to  break  the 
model  down  into  small  manageable  pieces. 

The  user  manuals  are  easy  to  follow.  There  is  an  index  in  the  back  of  each  book 
of  the  documentation.  The  index  provides  a  quick  reference  to  all  of  the  documentation 
manuals.  The  manuals  provide  details  and  execution  steps  for  most  of  the  topics. 

Once  again  run  time  and  set  up  time  are  dependent  on  the  detail  of  the  model. 
The  user  must  design  the  model  to  the  accuracy  desired  in  the  solution.  Workbench  uses 
multithread  processing  to  simulate  numerous  events  occurring  simultaneously.  The  run 
length  and  run  interval  time  or  clock  speed,  can  be  altered  prior  to  the  execution  of  the 
simulation. 

9.  Experimental  Design 

Workbench  does  not  offer  an  iterative  run  capability.  Workbench's  basic  features, 
modeling  with  sub-modules,  enables  the  user  to  model  emerging  system  by  breaking 
them  down  into  their  smallest  elements. 

10.  Extendibility 

Workbench  uses  a  C  based  language.  Users  can  add  C  language  procedures  to  be 
included  in  the  simulation  program.  These  additions  enable  Workbench  to  be  extended 
as  necessary.  The  user  can  access  the  code  developed  for  the  model. 

11.  Extensibility  Of  Results 

The  ability  of  the  user  to  add  C  programming  statements,  also,  enables  the  user  to 
create  new  and  alternative  computers,  communication  equipment  and  any  other  type  of 
system.  This  capability  is  limited  by  the  user's  ability  to  program  in  C  language. 

SES  offers,  for  a  price,  extension  products. 

12.  Fidelity 

This  product  seems  to  provide  realistic  results  consistent  with  the  input 
parameters  and  model.  However,  Workbench  is  graph-oriented  programming,  which  is 
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dependent  on  user  definitions  for  its  behavior.  Models  can  be  developed  with  various 
levels  of  detail. 

13.  Flexibility 

Nodes  can  be  delayed  or  blocked,  simulating  movement  or  destruction,  through  a 
time  activation,  probability  distribution,  or  a  trigger.  Each  of  these  must  be  created  as 
part  of  the  model. 

14.  Interoperability 

Workbench  can  intemperate  with  most  programs  created  in  C.  C  programs  can  be 
imported  into  the  model  and  run  as  part  of  the  simulation. 

15.  Miscellaneous 

Workbench  does  not  offer  any  terrain  or  frequency  features. 

Outages  must  be  developed  as  part  of  the  model.  Delays  and  blocks  can  simulate 
temporary  and  permanent  outages.  Time  delays,  probability  distributions  and  triggers  can 
be  established  to  create  and  fix  outages. 

16.  Operational  Environment 

Workbench  is  only  available  for  UNIX  platforms,  and  thus  are  not  very  portable. 
The  platforms  are  commercially  available. 

17.  Post  Analysis  Support 

Workbench  provides  textual  reports  and  machine  readable  output.  The  reports 
can  be  displayed  on  the  screen  or  printed.  The  reports  consist  of:  a  cover  page,  two 
optional  user-generated  output  sections,  a  report  header,  a  detailed  statistical  report,  an 
optional  node  summary  statistical  report,  and  an  ending  banner.  The  machine  readable 
output  can  be  post  processed  using  most  commercial  spreadsheets  or  word  processors. 
The  user  can,  also,  print  the  graphical  representation  of  the  model. 

Available  only  on  SUN  platforms,  SES/graph  is  a  small  easy  to  use  graph 
program.  It  is  a  front  end  for  a  program  called  grafit R  created  by  Graphicus,  Inc., 
Kirkland,  Washington.  (SES,  1994-1995,  pp.  A-l).  SES/graph  enables  the  user  to 
display  and  plot  statistical  data  produced  during  the  model's  run.  These  graphs  can  be 
displayed  in  bar  charts,  pie  charts,  line  charts,  spline  charts,  or  a  step  function  chart. 
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18.  Reliability 

The  product  executes  in  its  own  run  time  environment.  Debugging  features  are 
included. 

19.  Security 

Workbench  does  not  offer  any  security  features. 

20.  Software 

Workbench  is  based  on  the  C  language.  Workbench  uses  an  object-oriented 
environment.  The  user  can  access  and  alter  the  code  developed  for  the  model. 

21.  Training  /  Tutorial 

Workbench  training  is  available  at  SES  Regional  Sites.  Customized  training  can 
be  requested,  for  a  cost,  to  be  conducted  at  the  user's  site.  Introductory  and  advance 
classes  are  available. 

Workbench's  documentation  includes  a  basic  tutorial.  It  is  a  quick  overview  of 
the  program's  primary  features. 

22.  Troubleshooting  Assistance 

Workbench  has  animation  and  trace  capabilities.  The  user  can  view  the  code  by 
accessing  the  model's  file. 

SES  offers  assistance  via  a  1-800  telephone  number,  email,  the  world  wide  web, 
and  through  regional  sites.  Frequently  asked  questions  are  available  through  the  world 
wide  web  to  registered  users. 

23.  User  Interface 

The  user  interface  is  graphic  oriented.  The  user  through  menus,  icons,  and  arcs  or 
connection  tools,  creates  a  graphic  picture  of  the  model.  The  flow  of  the  menus  can  be 
confusing  to  the  novice  user.  The  user  can  modify  the  parameters  of  the  nodes  and  can 
add  C  programming  to  augment  the  simulation. 

24.  Validation 

Workbench  provides  the  user  with  the  option  of  modeling  to  the  small  details  or 
on  a  larger  scale.  Models  programmed  via  code  will  depend  on  the  users  modeling  ability 
for  validity. 
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25.       Verification 

No  verification  features  were  identified.  The  product  seems  to  function  as 
described  in  the  manual. 

C.         CONCLUSIONS 

The  basic  commands  for  Workbench  are  easy  to  learn.  Workbench  requires  the 
user  to  be  a  proficient  modeler.  The  user  must  be  able  to  understand  the  basic  functions 
of  the  system  they  are  modeling.  Workbench,  also,  places  heavy  emphasis  on 
understanding  programming.  Workbench  is  a  good  tool,  however,  it  is  not  designed  for 
the  beginner. 
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VI.       G2 

A.  INTRODUCTION 

G2,  from  the  Gensym  Corporation,  is  designed  for  developing  intelligent, 
knowledge-based,  real-time  applications.  G2  is  typically  utilized  for  complex  situations 
that  require:  monitoring,  diagnosis,  and  alarm  handling;  scheduling  and  logistics; 
supervisory  and  advanced  control;  process  design,  simulation,  and  re-engineering; 
intelligent  network  management;  and  decision  support  for  enterprise-wide  operations 
(Gensym,  1995,  pp.  4). 

Each  intelligent  application  developed  or  modeled  is  known  as  a  knowledge  base 
(KB).  KBs  are  developed  in  a  graphical  and  menu  oriented  environment  using  Object 
Oriented  Modeling.  Gensym  Corporation  uses  Object  Oriented  Modeling  as  the  bases  of 
the  G2  product. 

B.  ASSESSMENT  CRITERIA 

1.  Accessibility 

G2  can  be  installed  on  UNIX,  VMS,  Windows  95,  and  Windows  NT  platforms. 
These  include  Digital  Equipment  Corporation,  Hewlett  Packard,  Sun  SPARC  stations, 
Silicon  Graphics,  and  IBM  Reduced  Instruction  Set  Computers.  G2  requires  30  MB  of 
hard  disk  space  and  48  MB  of  RAM. 

2.  Completeness 

G2  enables  the  user  to  model  computers,  software,  and  communication  equipment 
through  the  use  of  Object  Oriented  Modeling.  Specific  KBs  must  be  developed  by  the 
user. 

G2  provides  the  modeler  with  the  capabilities  to  create  not  only  the  model  but  the 
interface  that  the  operator  will  have  with  the  simulation.  The  operator  can  be  prompted 
for  input  via  toggle  switches,  text  inputs,  or  interactive  buttons.  The  modeler  can  create 
an  interactive  on-line  simulation  environment. 

The  Army  Artificial  Intelligence  Center  is  currently  using  G2. 
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3.  Configuration  Management 

Gensym  Corporation  was  founded  in  1986.  Gensym  Corporation  currently 
provides  services  to  over  30  countries  which  include  petroleum  companies,  chemical 
companies  and  other  industrial  companies. 

The  user  is  responsible  for  managing  input  and  output  files.  G2  graphically  shows 
the  objects  within  the  designed  system.  Data  encapsulation  or  hiding  is  available  to 
reduce  clutter  and  access  to  some  information.  Object  relationships  can  be  depicted  by 
showing  the  workspace  module  hierarchy. 

4.  Cost 

The  current  commercial  cost  for  the  G2  server  license  with  Telewindows  is 
$36,000.00.  A  minimum  of  one  user  license  must  be  purchased  at  $4,800.00  (User  license 
prices  decrease  after  the  eleventh  user).  The  university  cost  is  $500.00.  The  GSA  cost  is 
3 1  %  of  the  commercial  cost. 

Customer  service  support  is  available  for  a  yearly  fee  of  15%  of  the  commercial 
list  price  ($6,120.00  for  a  single  user  package). 

Free  one  and  one  half  day  workshops  are  available  at  select  regional  sites.    G2 
Part  I  and  Part  II  training,  each  a  four  and  one  half  day  course,  is  available  for  $2,295.00 
each. 

5.  Data  Input  Methods 

G2  includes  internal  and  external  data  source,  or  data  server,  features.  G2's 
internal  data  servers  are  the  G2  inference  engine  and  the  G2  simulator.  The  G2  inference 
engine  "concludes"  or  calculates  an  answer  from  specific  formula  objects.  The  G2 
simulator  receives  a  value  from  a  simulation  formula. 

External  data  server  methods  include  data  received  from  other  G2  processes,  from 
GFI  bridge  applications,  and  from  GFI  files.  (See  Interoperability  sub-section  for  more 
information) 

6.  Data  Reduction 

G2  does  not  offer  any  standard  statistical  features. 
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7.  Documentation 

G2  provides  a  complex  assortment  of  documentation.  The  titles  of  the  manuals 
are  very  misleading  as  to  the  content  to  a  beginning  user.  These  manuals  include:  G2 
Reference  Manual,  Upgrading  Your  Knowledge  Bases,  G2  Systems  Procedures  Reference 
Manual,  Telewindows  User's  Guide  ,and  G2  GUIDE  User's  Guide.  G2  documentation 
is  very  complicated.  It  is  oriented  toward  a  proficient  object  oriented  modeler. 

G2  does  not  offer  any  tutorial. 

Design  and  modeling  documentation  is  not  available. 

8.  Ease  Of  Use 

G2  is  not  easy  to  learn.  It  is  entirely  object  oriented.  Existing  KBs  are  not  labeled 
well  enough  to  know  what  they  contain.  The  user  can  create  KBs  of  computers  and 
communication  equipment  and  save  them  for  reuse. 

The  manuals  provide  guidance  on  the  G2  functions  and  features  but  do  not  cover 
any  Object  Oriented  Modeling. 

Set  up  time  is  dependent  on  the  detail  of  the  model,  the  related  KBs  available  to 
build  upon,  and  the  users  ability  to  use  object  oriented  analysis  and  design. 

G2  has  three  options  for  running  a  simulation:  real  time,  simulated  time,  and  as 
fast  as  possible.  The  model  and  the  inputs  can  be  altered  while  the  simulation  is  running. 

9.  Experimental  Design 

G2  does  not  offer  any  iterative  run  capabilities.  Object  oriented  modeling  enables 
the  user  to  create  objects  and  attributes  to  model  emerging  systems. 

10.  Extendibility 

G2  models  can  be  altered  on  line  or  off  line.  G2  uses  a  structured  natural 
language,  similar  to  COBOL.  The  language  uses  ordinary  human  statements,  so  the  user 
does  not  need  to  be  a  proficient  programmer.  G2's  natural  language,  also,  supports 
Japanese,  Korean,  and  Russian. 

A  text  editor  is  available  for  the  user  to  input  specific  statements,  rules,  functions, 
attributes,  etc. 


39 


11.  Extensibility  Of  Results 

G2  and  Object  Orient  Modeling  provide  the  user  with  a  flexible  environment 
capable  of  modeling  virtually  any  computer  or  communication  system  the  user  can 
imagine.  The  product  is  limited  by  the  users  ability  to  analyze  and  design  a  model.. 

12.  Fidelity 

The  tool  reliably  provides  results  consistent  with  the  input  parameters.  Models 
can  be  developed  with  various  degrees  of  detail. 

13.  Flexibility 

Nodes  can  be  moved  both  on  and  off-line.  The  user  can  establish  time  delays, 
triggers  or  interactive  inputs  to  initiate  the  move  or  outage. 

14.  Interoperability 

G2  provides  networking  and  interfacing  through  Telewindows,  G2  to  G2 
Interface,  G2  Standard  Interface  (GSI),  and  G2  File  Interface(GFI).  Telewindows  allows 
more  than  one  user  to  access,  or  remote  view,  a  running  G2  process.  G2  to  G2  Interface 
lets  two  or  more  G2  process  to  connect  to  share  data.  GSI  is  a  toolkit  for  developing 
bridges  between  G2  and  external  systems.  GFI  allows  G2  to  read  and  write  external  data 
files. 

15.  Miscellaneous 

Outages  can  be  simulated  with  time  delays,  triggers,  or  interactive  input  by  the 
operator. 

Frequency  and  terrain  options  are  not  available  with  G2. 

16.  Operational  Environment 

This  product  is  available  for  UNIX,  VMS,  and  Windows  platforms.  All  these 
systems  are  COTs  and  easily  obtainable  to  users.  Portability  depends  on  the  platform  the 
software  is  installed  on. 

17.  Post  Analysis  Support 

G2  features  include  readout  tables,  dials,  meters,  freeform  tables,  charts,  graphs 
and  trend  charts.  Readout  tables,  dials  and  meters  depict  changing  values  during  a 
simulation  run.  Freeform  tables  are  similar  to  spreadsheets.  Charts  plot  numerical  data 
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series.  Charts  include  data  points,  bar  charts  and  column  charts.  Graphs  are  being 
replaced  by  trend  charts.  Trend  charts  plot  times  series  and  historical  data. 

18.  Reliability 

The  product  executes  in  its  own  run  time  environment.  Results  are  consistent 
with  the  inputted  model. 

19.  Security 

G2  provides  network  security  features.  These  features  include:  connect,  read, 
write,  execute,  and  inform.  Connect  prohibits  or  enables  other  users  from  accessing  a 
KB.  Read  and  write  prohibit  or  enable  other  users  from  reading  or  writing  variable 
values.  The  execute  command  deals  with  remote  procedure  calls,  and  the  inform 
command  prohibits  or  enables  other  users  from  sending  messages  to  the  operator. 

20.  Software 

G2's  structured  natural  language  provides  the  modeler  with  a  user  friendly 
program  interface.  The  modeler  can  use  ordinary  human  language  statements  to  input 
specific  commands  for  the  model  to  execute.  G2's  language  and  modeling  environment 
are  object  oriented. 

21.  Training  /  Tutorial 

G2  offers  numerous  training  courses:  Introduction,  G2  Part  I,  G2  Part  n,  and  many 
classes  for  supporting  products.  Courses  provide  a  wide  variety  of  information  on  how  to 
use  G2.  The  combine  syllabus  for  the  Introduction,  and  G2  Parts  I  and  II  indicates  that 
upon  completion  of  the  training  the  user  will  have  an  intermediate  knowledge  of  the 
product. 

No  tutorial  is  available.  Some  demonstration  KBs  are  included. 

22.  Troubleshooting  Assistance 

G2  has  been  rated  number  one  in  customer  support  by  Control  Magazine  for  the 
last  two  years  (Gensym,  1997,  pp.  "Services").  Engineers  are  available,  by  telephone, 
Monday  through  Friday,  to  assist  with  installation,  product  use,  problem  diagnosis,  and 
configuration  questions. 
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G2  Help  Link  is  a  24  hours  a  day,  seven  days  a  week,  automated  help  service 
available  through  the  G2  world  wide  web  home  page.  The  Gensym  Users  Society  (GUS) 
provides  technical  communications  between  Gensym  customers  and  the  Gensym 
Corporation.  Help  services  are,  also,  available  through  email. 

Animation  is  not  available. 

23.  User  Interface 

The  G2  user  interface  is  menu  oriented.  The  number  of  menus  can  be 
overwhelming  to  the  beginning  user.  Menus  lead  to  construction  of  objects,  classes, 
connections,  etc.  Once  an  item  is  created,  it  is  defined  through  a  table.  Most  table  entries 
are  prompted  with  a  list  of  available  options.  Some  entries  allow  the  user  to  provide 
input  into  the  model.  For  example:  variable  and  object  names,  or  the  use  of  structured 
natural  language  to  input  exact  specifications  for  an  object. 

24.  Validation 

Users  can  model  to  the  depth  desired.  The  user  must  use  correct  object  oriented 
analysis  to  properly  define  the  system  to  be  modeled. 

25.  Verification 

The  product  functioned  as  described  in  the  manuals.  Gensym  Corporation  has 
sold  more  than  7500  product  licenses  to  over  500  organizations  worldwide  (Gensym, 
1997,  pp.  "Company"). 

C.        CONCLUSIONS 

G2  is  based  on  object  oriented  analysis  and  design.  "Object  oriented  analysis  is 
based  upon  concepts  that  we  first  learned  in  kindergarten:  objects  and  attributes,  classes 
and  members,  wholes  and  parts."  (Pressman,  1992,  pp.  239) 

The  product,  while  initially  intimidating,  is  a  very  powerful  tool.  The  structured 
natural  language  enables  a  non-programmer  to  have  the  features  and  capabilities  available 
to  create  a  detailed  model  of  a  complex  system. 
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VII.      ANALYSIS 

A.        INTRODUCTION 

This  section  compares  the  five  software  products.  A  rating  of  1  to  5  has  been 
established  for  each  evaluation  criteria  based  on  the  sub-criteria  contained  in  the 
Appendix: 

1  -  no  sub-criteria  satisfied 

2  -  less  than  half  of  the  sub-criteria  satisfied 

3  -  half  of  the  sub-criteria  satisfied 

4  -  more  than  half  of  the  sub-criteria  satisfied 

5  -  all  of  the  sub-criteria  satisfied. 

Additionally,  general  usage,  vendor,  available  platforms  and  the  average  price  have  been 
included. 


B. 


COMPARISON 


Table  7-1  shows  a  non- weighted  comparison  of  the  software  products. 
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Simulation 

Time  Systems 

Computer 

LAN/WAN 

Tool 

Systems  and 

Applications, 

and  Distributed 

Systems 

Vendor 

MIL  3  Inc. 

CACI  Products 
Company 

Imagine  That, 
Inc. 

SES 

Gensym  Corp. 

Platforms 

Sun  SPARC, 

PCs  (Windows 

PCs  (Windows 

Sun  Micro- 

Digital,  IBM, 

DEC,  HP  UX, 

NT  and 

3.1,  NT  and 

stations,  IBM, 

HP,  Sun 

SGI,  Windows 

Windows  95), 

Windows  95), 

HP 

Micro-stations, 

NT 

UNIX,  DEC, 
HP  UX,  SGI, 
AIX,  Solaris 

Macintosh 

PC's 

(Windows  NT 
and  Windows 
95) 

Price 

Commercial 

$18,000.00 

$48,500.00 

$695.00 

$23,500.00+ 

$40,800.00 

University 

$600.00 

$3000.00 

$350.00 

$500.00 

$500.00 

Accessibility 

3 

3 

3 

2 

3 

Completeness 

5 

3 

4 

4 

4 

Configuration 

5 

4 

4 

5 

5 

Management 

Cost 

4 

1 

5 

3 

2 
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OPNET 
MODELER 

COMNET  m 

EXTEND 

WORKBENCH 

G2 

Data  Input 
Methods 

5 

5 

5 

5 

5 

Data 
Reduction 

5 

5 

5 

5 

3 

Documentation 

4 

3 

3 

4 

3 

Ease  Of  Use 

3 

4 

3 

3 

2 

Experimental 
Design 

4 

2 

5 

3 

3 

Extendi  bility 

4 

2 

4 

5 

5 

Extensibility 
Of  Results 

5 

2 

4 

4 

4 

Fidelity 

5 

4 

5 

5 

5 

Flexibility 

4 

4 

4 

4 

4 

Interoperability 

4 

2 

4 

4 

4 

Miscellaneous 

4 

2 

2 

2 

2 

Operational 
Environment 

5 

5 

5 

3 

5 

Post  Analysis 
Support 

5 

5 

3 

4 

5 

Reliability 

5 

5 

5 

5 

5 

Security 

3 

1 

1 

1 

3 

Software 

3 

3 

3 

4 

4 

Training  / 
Tutorial 

4 

4 

4 

3 

2 

Troubleshooting 
Assistance 

3 

3 

4 

5 

3 

User  Interface 

4 

5 

5 

4 

4 

Validation 

5 

5 

5 

5 

5 

Verification 

5 

5 

5 

5 

5 

Non  Weighted 
Totals 

106 

87 

100 

97 

95 

Table  7-1.  Comparison  of  Products. 

C.        THE  GRAND  SOLUTION 

A  quick  total  suggests  that  OPNET  is  the  best  overall  software  product.  However, 
the  criteria  must  be  weighted  when  selecting  a  specific  software  product  for  the  user's 
needs. 

By  weighting  each  evaluation  criterion,  a  priority  for  the  characteristics  required 
for  a  mission  is  established.  These  weights  may  differ  for  each  distinct  task.  An 
organization  should  give  serious  consideration  to  the  weights  based  on  their  generic  usage 
of  the  product  (unless  the  organization  can  afford  to  purchase  more  than  one  product). 
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This  will  provide  the  organization  with  a  product  that  may  not  be  the  "best"  for  every 
instance,  but  will  be  the  "best"  for  the  organization's  overall  usage. 

An  example,  a  unit  is  looking  for  a  software  product  that  can  model  experimental 
architectures  and  systems  in  addition  to  communication  and  computer  networks.  The  unit 
has  no  UNIX  machines,  mostly  PCs,  and  a  few  Macintoshes.  The  operators  are  proficient 
in  C  Language,  but  troubleshooting  assistance  and  user  interface  are  considered  key  for 
the  experimental  design  of  the  system.  Frequency  and  terrain  have  no  bearing  on  the 
systems  to  be  modeled.  Minimizing  cost  is  very  important  to  the  unit.  The  unit  may 
establish  the  following  weights:  Completeness  -  1,  Cost  -  5,  Experimental  Design  -  5, 
Extendibility  -  4,  Miscellaneous  -  1,  Security  -  1,  Software  -  5,  Troubleshooting 
Assistance  -  5,  User  Interface  -  5,  and  all  other  criteria  may  be  a  3.  Each  weight  is  then 
multiplied  by  the  product's  score  to  produce  a  weighted  score.  The  weighted  scores  are 
totaled.  The  product  with  the  highest  weighted  score  is  the  best  modulation  simulator 
based  on  the  current  priorities.  In  this  example  that  would  be  Extend.  Table  7-2  depicts 
the  comparison  example  with  weighted  scores. 


OPNET 
MODELER 

COMNET  m 

EXTEND 

WORKBENCH 

G2 

General  Use 

5*3  =  15 

5*1=5 

5*4  =  20 

5*4  =  20 

5*4  =  20 

Platform 

4*4  =  16 

4*4  =  16 

4*5  =  20 

4*1=4 

4*4  =  16 

Accessibility 

3*3  =  9 

3*3  =  9 

3*3  =  9 

3*2  =  6 

3*3  =  9 

Completeness 

1*5  =  5 

1*3  =  3 

1*4  =  4 

1*4  =  4 

1*4  =  4 

Configuration 
Management 

3*5  =  15 

3*4=12 

3*4  =  12 

3*5  =  15 

3*5  =  15 

Cost 

5*4  =  20 

5*1=5 

5*5  =  25 

5*3=15 

5*2=10 

Data  Input 
Methods 

3*5  =  15 

3*5  =  15 

3*5  =  15 

3*5  =  15 

3*5  =  15 

Data 
Reduction 

3*5  =  15 

3*5  =  15 

3*5  =  15 

3*5  =  15 

3*3  =  9 

Documentation 

3*4=  12 

3*3  =  9 

3*3  =  9 

3*4  =  12 

3*3  =  9 

Ease  Of  Use 

3*3  =  9 

3*4=12 

3*3  =  9 

3*3  =  9 

3*2=6 

Experimental 
Design 

5*4  =20 

5*2=  10 

5*5  =25 

5*3=15 

5*3=15 

Extendibility 

4*4=16 

4*2  =  8 

4*4=16 

4*5  =  20 

4*5  =  20 

Extensibility 
Of  Results 

3*5  =  15 

3*2  =  6 

3*4=12 

3*4=  12 

3*4=12 

Fideuty 

3*5  =  15 

3*4=12 

3*5  =  15 

3*5  =  15 

3*5  =  15 

Flexibility 

5*4  =  20 

5*4  =  20 

5*4  =  20 

5*4  =  20 

5*4  =  20 

Interoperability 

2*4  =  8 

2*2  =  4 

2*4  =  8 

2*4  =  8 

2*4  =  8 

Miscellaneous 

1*4  =  4 

1*2  =  2 

1*2  =  2 

1*2  =  2 

1*2  =  2 

Operational 
Environment 

3*5  =  15 

3*5  =  15 

3*5  =  15 

3*3  =  9 

3*5  =  15 
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OPNET 
MODELER 

COMNET  m 

EXTEND 

WORKBENCH 

G2 

Post  Analysis 
Support 

3*5  =  15 

3*5=  15 

3*3  =  9 

3*4=12 

3*5  =  15 

Reliability 

3*5-  15 

3*5=15 

3*5  =  15 

3*5=  15 

3*5  =  15 

Security 

1*3  =  3 

1*1  =  1 

1*1  =  1 

1*1  =  1 

1*3  =  3 

Software 

5*3=15 

5*3=15 

5*3=15 

5*4  =  20 

5*4  =  20 

Training  / 
Tutorial 

3*4=  12 

3*4=  12 

3*4=  12 

3*3  =  9 

3*2  =  6 

Troubleshooting 
Assistance 

5*3=15 

5*3=15 

5*4  =  20 

5*5  =  25 

5*3=15 

User  Interface 

5*4  =  20 

5*5  =  25 

5*5  =  25 

5*4  =  20 

5*4  =  20 

Validation 

3*5=15 

3*5  =  15 

3*5  =  15 

3*5  =  15 

3*5=15 

Verification 

3*5  =  15 

3*5  =  15 

3*5  =  15 

3*5  =  15 

3*5  =  15 

Non  Weighted 
Totals 

369 

306 

378 

348 

344 

Table  7-2.  Example  Weighted  Comparison  of  Products. 

D.         OVERALL  COMPARISON 

Each  of  the  five  products  analyzed  in  this  thesis  has  its  own  special  characteristics 
and  capabilities.  Why  select  one  over  the  other?  The  functions  and  capabilities  vary 
making  some  products  more  preferable  for  different  applications  and  scenarios. 

CACI's  COMNET  m  is  the  best  choice  for  modeling  a  WAN  or  LAN.  COMNET 
lH  provides  all  the  familiar  computer  network  icons  in  a  user  friendly  environment  to 
enable  the  user  to  quickly  model  and  analyze  a  computer  network.  COMNET  HI, 
however,  does  not  model  communication  or  information  systems.  Nor  does  COMNET  IQ 
model  the  innards  of  a  computer. 

MIL  3  developed  OPNET  Modeler  which  can  model  computers,  communication 
and  information  systems  from  the  network  through  the  process  and  state  levels. 
Numerous  libraries  are  available  to  assist  the  modeler.  OPNET' s  primary  disadvantage  is 
its  fixed  run  speed.  This  can  cause  extensive  time  delays  during  a  simulation  run  of  a 
detailed  model. 

SES'  Workbench,  like  OPNET,  can  be  used  to  model  communication,  computer, 
and  information  systems.  In  addition,  Workbench  can  be  used  to  model  numerous  other 
types  of  systems,  to  include  queues,  electric  flow,  water  pollution,  etc..    The  ability  to 
model  more  than  just  communication,  computer  and  information  systems  provides  an 
added  flexibility  to  the  modeler. 
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Extend,  developed  by  Imagine  That  Inc.,  is  similar  in  its  capabilities  to 
Workbench,  in  that  it  can,  also,  model  any  type  of  system.  Extend,  however,  provides  a 
more  user  friendly  environment  for  the  modeler.  Besides  the  documentation's  ease  of 
use,  the  menu  setup  has  a  more  logical  flow.  Workbench,  through  the  use  of  sub- 
modules,  can  model  to  a  smaller  level  of  detail  than  Extend,  and  Workbench  can  handle 
larger  sized  models. 

G2,  also  similar  to  Workbench  in  its  system  modeling  capabilities,  is  in  a  class  by 
itself.  G2  utilizes  an  object  oriented  modeling  environment  with  a  structured  natural 
language.  This  product  is  a  great  selection  for  a  beginner  or  non-programmer.  Object 
oriented  modeling  enables  the  modeler  to  design  the  level  of  fidelity  and  complexity  to  be 
modeled.  G2  includes  a  wide  range  of  knowledge  bases  or  libraries,  though  the  labeling 
is  puzzling.  G2/object  oriented  modeling  is  not  an  easy  product/modeling  method  to 
learn. 

It  is  essential  to  look  at  the  capabilities  and  characteristics  of  the  modeler(s). 
Programming  skills  can  influence  the  product  selection.  All  five  of  the  products  are  icon 
and  menu  oriented.  COMNET  HI  provides  the  smallest  programming  interface,  and  G2 
has  an  easy  to  learn  structured  natural  language.  OPNET,  Extend,  and  Workbench,  each 
use  a  modified  version  of  the  C  language.  Workbench  presents  the  user  with  the  largest 
interface  with  the  model's  code,  and  provides  the  user  with  the  most  programming 
opportunities. 
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VIII.    CONCLUSIONS 

A.         SUMMARY 

Today's  technological  advances  are  nearly  overwhelming.  Increasing  bandwidths, 
increasing  data  rates,  new  protocols  and  more,  all  lead  to  new  equipment  available  for  the 
military.  The  increased  usage  of  COTS  products  encourages  units  to  upgrade  and  replace 
existing  capabilities. 

The  software  products,  analyzed  in  this  thesis,  can  provide  the  user  with  a  method 
for  evaluating  new  equipment  prior  to  its  purchase.  It,  also,  enables  the  user  to  optimize 
existing  systems  and  networks,  and  to  evaluate  "what  if  situations. 

Each  product  has  individual  characteristics  which  make  it  the  best  choice  for  a 
particular  user  in  a  specific  situation.  This  thesis  evaluated  five  modulation  simulator 
software  products  based  on  twenty-five  evaluation  criteria. 

OPNET  Modeler,  by  MIL  3  Inc.,  provides  built-in  familiar  network  icons.  It 
provides  the  user  with  limited  access  to  the  code.  The  user  can  program  entrance  and  exit 
criteria  for  low  level  states  and  processes.  Access  to  code  and  debugging  is  in  Proto-C. 
This  can  be  troublesome  for  a  non-programmer. 

COMNET  m,  from  CACI  Products  Company,  is  easy  to  learn.  It  provides  quick 
high  to  intermediate  level  network  modeling  with  familiar  built-in  icons. 

Extend,  developed  by  Imagine  That  Inc.,  is  very  user  friendly.  It  does  not  contain 
network  icons,  but  the  user  can  build  them.  The  product  is  flexible  and  can  be  used  to 
model  more  than  just  networks,  computers  and  communication  equipment. 

Workbench,  created  by  Scientific  and  Engineering  Software  Inc.,  permits  users  to 
access  the  model's  code.  Network  icons  are  not  included,  but  can  be  created  by  the  user 
or  purchased  for  an  additional  cost.  Access  to  the  model's  code  and  debugging  the  model 
are  all  completed  in  a  language  similar  to  C.  These  can  be  difficult  for  a  non- 
programmer. 

G2,  by  Gensym  Corporation,  provides  a  completely  object  oriented  modeling 
environment.  The  documentation  is  very  confusing.  Programming  is  easy  for  non- 
programmers.  It  is  completed  in  a  structured  natural  language  similar  to  the  human 
language. 
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A  user's  needs  and  modeling  and  programming  capabilities  should  be  evaluated 
prior  to  choosing  a  software  product.  A  weighted  criteria  evaluation  is  recommended  for 
selecting  the  correct  software  tool. 

B.  APPLICATIONS  WITHIN  THE  MILITARY 

The  complex  technology  and  the  limited  availability  of  troops  and  money 
provides  an  environment  without  the  capability  to  fully  test,  evaluate,  integrate,  and 
analyze  new  equipment.  Modulation  simulators  can  be  used  to  evaluate  the  integration  of 
new  equipment.  New  equipment  can  be  modeled,  tested  and  integrated  for  less  money 
and  with  less  troops  than  physically  fielding  a  piece  of  equipment.  Existing  systems  can 
be  load-tested,  reorganized,  and  reconnected.  Outages  and  equipment  losses  can  be 
simulated  without  the  use  of  actual  equipment  and  soldiers. 

C.  RECOMMENDED  FUTURE  STUDIES 

The  software  tools  evaluated  in  this  thesis  can  be  used  to  model  virtually  any 
computer,  communication,  or  information  system.  Future  studies  could  include  modeling 
and  evaluating  existing  or  emerging  command  and  control  or  communications  systems. 

The  growing  inventory  of  modulation  simulators  has  produced  a  concern  for 
interface  capabilities  and  limitations.  This  can  be  expanded  to  include  uses  and  interfaces 
with  virtual  reality  simulators. 

Additionally,  the  Naval  Postgraduate  School  is  upgrading  its  existing  internal 
computer  network.  The  current  and  future  networks  could  be  modeled  and  analyzed  for 
shortcomings  and  improvements. 
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APPENDIX.  DEFINITIONS  OF  ASSESSMENT  MEASURES 

1.  ACCESSIBILITY 

•  Platforms  supported 

•  Conversion  packages  for  switching  to  other  platforms 

•  Plans  for  support  to  any  other  platforms  in  the  near  future 

•  PC  version:  DOS  or  UNIX  based? 

2.  COMPLETENESS 

•  Interactively  models  computers,  software  and  communications 
equipment 

•  Includes  or  is  capable  of  modeling  current  and  new  computer 
architectures 

•  Includes  or  is  capable  of  modeling  current  and  new  LANs  /  WANs 

•  Includes  or  is  capable  of  modeling  current  or  new  mass  storage  devices 

•  Includes  or  is  capable  of  modeling  Intra-  and  Inter-  machine 
communications 

•  Includes  or  is  capable  of  modeling  military  communication  devices 

3.  CONFIGURATION  MANAGEMENT 

•  Maturity  of  software 

•  User  view  of  the  model 

4.  COST 

•  Military  acquisition  prices  (What  extras  are  included?) 

•  Maintenance  costs 

•  Training  costs 

•  Upgrade  costs 

5.  DATA  INPUT  METHODS 

•  Internal  Methods 

•  External  Methods 

6.  DATA  REDUCTION 

•  How  quickly 
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•  What  methods  (Poisson,  Wiebull,  etc.) 

7.  DOCUMENTATION 

•  Requirements  specification 

•  Design  document 

•  Conceptual  model  document 

•  Tutorial 

•  User's  Manual 

8.  EASE  OF  USE 

•  Support  for  computer/network  modeling 

•  Support  for  software  modeling  (pre-analysis  sizing,  communications, 
functional  flow,  data  flow,  CPU  loading,  machine  instruction 
specification,  etc.) 

•  Support  for  data  channeling  modeling 

•  Support  for  communications  traffic  modeling 

•  Ease  of  learning 

•  Manuals  readily  available  and  easy  to  understand 

•  Set  up  time 

•  Run  time 

*  Can  it  be  varied 

*  Typical  run  duration 

9.  EXPERIMENTAL  DESIGN 

•  Supports  Monte  Carlo 

•  Supports  algorithms 

•  Iterative  runs 

10.  EXTENDIBILITY 

•  Can  tool  be  readily  modified? 

•  Design  or  programmatic? 

•  What  language? 

11.  EXTENSIBILITY  OF  RESULTS 
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•  Alternative  computer  architectures 

*  Extension  to  hypothetical  architectures 

*  Extension  to  architectures  that  may  be  performance  tested 

*  Extension  to  architecture  that  are  not  available  for  testing 

•  Alternative  data  channels 

*  Extension  to  hypothetical  architectures 

*  Extension  to  architectures  that  may  be  performance  tested 

*  Extension  to  architecture  that  are  not  available  for  testing 

•  Alternative  software  descriptions 

•  Alternative  communication  requirements 

12.  FIDELITY 

•  Tool  provides  sufficient  fidelity  to  ensure  output  accuracy  across  the 
tool  architecture 

13.  FLEXIBILITY 

•  Can  nodes  move  on  line  /  off  line?  (How  implemented) 

•  How  quickly  reconfigure? 

14.  INTEROPERABILITY 

•  Known  interface  capabilities  (software  /  hardware) 

•  Interfaces  under  development 

•  Partial  interfaces  available  (just  the  scenario?) 

15.  MISCELLANEOUS 

•  Terrain 

*  Is  it  available? 

*  Can  equipment  be  tied  to  terrain? 

•  Frequency  Management  capabilities 

•  Outages 

*  Planned  /  unplanned  outages  (How  implemented?) 

•  Additional  features 

16.  OPERATIONAL  ENVIRONMENT 
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•  System 

•  Portable 

•  Availability 

17.  POST  ANALYSIS  SUPPORT 

•  Reports  (Variety) 

•  How  quickly 

•  Graphs  (Variety) 

18.  RELIABILITY 

•  Will  it  work  every  time  producing  the  same  results? 

19.  SECURITY 

•  Type  of  security  (Classified  or  User) 

•  Labeling 

•  Passwords 

20.  SOFTWARE 

•  Process  maturity 

•  Architecture 

•  Complexity 

•  Quality 

•  Reliability 

•  Sufficiency  of  testing 

•  Modularity  or  Object  Oriented  Design 

•  Maintainability 

2 1 .  TRAINING  /  TUTORIAL 

•  Is  it  understandable?  User  friendly? 

•  Is  it  complete? 

•  How  proficient  is  user  after  training?  (overview,  beginner,  master) 

22.  TROUBLESHOOTING  ASSISTANCE 

•  Is  animation  available 

•  Ability  to  view  code  (Can  you  see  inside  the  model?) 
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•  1-800  help  lines 

•  Frequent  problems  list  /  lessons  learned 

23.  USER  INTERFACE 

•  Type  (Windows,  C  programming,  UNIX  programming,  etc.) 

•  Quality 

•  Effectiveness 

24.  VALIDATION 

•  Degree  of  accuracy 

•  Variable  (quick  answer,  intermediate  answer,  detailed  answer) 

25.  VERIFICATION 

•  Of  tool  design  and  functions  (How  do  we  know  it  works?) 
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